Agent frameworks & infrastructure
Model serving & inference
775 agents · most starred first
Model serving & inference, page 6 of 8
OpenLLM: Self-hosting LLMs Made Easy.
OpenLLM Client: Interacting with OpenLLM HTTP/gRPC server, or any BentoML server.
OpenLLM Core: Core components for OpenLLM.
OpenAI Compatible API Server using OpenVINO GenAI
轻量级大模型推理工具,专注于模型推理延迟,注重框架易用性和可拓展性。
One-line vLLM wrapper with gorgeous DSPy integration
Paragon AI — Python bindings (ctypes) with embedded cross-vendor GPU backends
High-performance key-value storage engine with Python bindings
High-performance key-value storage engine with Python bindings
Standalone MLX-based LLM inference service with OpenAI compatible API
Network Storage API
🌟 The Ultimate Multi-Model LLM Runtime Platform - Deploy, manage, and serve 300+ language models with OpenAI-compatible APIs. Built on ms-swift for production-ready performance.
Best collective comms library in the solar system
PrismLLM: Hardware-agnostic LLM inference library
ProcessFork plugin for vLLM ≥0.10 — paged-KV-cache snapshot/restore via batch-invariant kernels.
libLLM is an open-source project tailored for efficient inference of large language models (LLMs).
🦙 LLaMA: Open and Efficient Foundation Language Models in A Single GPU
Python wrapper for llama.cpp server - OpenAI API compatible backend with auto device detection
A general x-bit quantization engine for LLMs,[2-8] bits, awq/gptq/hqq/vptq
vLLM plugin for Qwerky AI MambaInLlama hybrid models
An open-source cloud-native of large multi-modal models (LMMs) serving framework.
Scale LLM Engine Python client
A high-performance inference system for large language models.
Dragon: A Computation Graph Virtual Machine Based Deep Learning Framework
Sentry File System Node Storage
Sentry Riak Node Storage
A simple description of your package
A simple description of your package
A middleware for model serving to speedup online inference.
Shard Large Language Models
A small example package
Sinapsis templates for text, batch, streaming, and multimodal generation with vLLM.
From-scratch paged-attention inference engine: paged KV cache, continuous batching, preemption
Sovereign Pulse physics gate — vLLM LogitsProcessor plugin
squeezellm
Community vLLM provider utilities for Strands Agents (OpenAI-compatible).
Find the breaking point of your local LLM hardware.
Serverless-native LLM orchestration framework for AWS Lambda
Superduper allows users to work with self-hosted LLM models via [vLLM](https://github.com/vllm-project/vllm).
serve your llm
Sych LLM Playground is a command line tool deploy and interact with language models on the cloud.
A high-throughput and memory-efficient inference and serving engine for LLMs
TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs.
The fork primitive for LLM inference. Snapshot a running session — weights + KV cache + scheduler state — and hydrate it into N divergent children that skip prefill. For RL rollouts, parallel coding agents, agent branching. Supports vLLM and SGLang.
A modular framework for LLM quantization, structured pruning, and edge deployment
Name reserved for the tokenspeed-trtllm-attn project.
Name reserved for the tokenspeed-trtllm-common project.
Name reserved for the tokenspeed-trtllm-gemm project.
Standalone TensorRT-LLM CUDA kernels as PyTorch custom ops
Name reserved for the tokenspeed-trtllm-moe project.
CUDA Kernels for Large Language Models (LLMs), compatible with torch.compile.
LLM and PEFT export to NNEF via torch-to-nnef.
An OpenAI Completions API compatible server for NLP transformers models
Flexible and modular LLM inference for mini-batch
Name reserved for the trtllm-kernel project.
Independent TurboQuant-style compression infrastructure for KV cache and RAG.
Reference implementation of TurboQuant-style vector/KV quantization (rotation + scalar quant + QJL residual).
TurboQuant+ compression for vLLM. 4.3x weight compression + 3.7x KV cache, zero calibration.
TurboQuant KV cache compression for vLLM — fused Triton kernels, 3.76x compression, 3.7x faster decode on RTX 4090
Type annotations for boto3 BedrockRuntime 1.44.0 service generated with mypy-boto3-builder 8.12.0
HTTP API bindings for llama.cpp built-in example server
Vineyard llm kv cache
Vital LLM Cluster Manager
A utility package for text generation using vLLM with multiprocessing support.
A high-throughput and memory-efficient inference and serving engine for LLMs
A high-throughput and memory-efficient inference and serving engine for LLMs
vLLM Allocator Adaptor (C/CUDA/Python) using callback shims
vLLM Ascend backend plugin
Block implementation of vLLM
gRPC client for vllm-bootstrap inference service
vLLM plugin: out-of-tree registration of canon-layer architectures (e.g. LlamaCanonForCausalLM from PhysicsLM4)
A CLI tool to conveniently serve LLMs with vLLM
Client for the vLLM API with minimal dependencies
Deploy, manage, and monitor vLLM instances across a GPU cluster from a single web dashboard.
A high-throughput and memory-efficient inference and serving engine for LLMs
A high-throughput and memory-efficient inference and serving engine for LLMs
A high-throughput and memory-efficient inference and serving engine for LLMs
vLLM CPU inference engine (AVX512 + VNNI + BF16 + AMX optimized)
vLLM CPU inference engine (AVX512 optimized)
vLLM CPU inference engine (AVX512 + VNNI + BF16 optimized)
vLLM CPU inference engine (AVX512 + VNNI optimized)
Dfloat11 plugin for vLLM
A high-throughput and memory-efficient inference and serving engine for LLMs
The LEGO set for custom vLLM model plugins — build, test, and deploy custom encoders, poolers, and kernels
A high-throughput and memory-efficient inference and serving engine for LLMs
Forward-only flash-attn
Out-of-tree GGUF quantization plugin for vLLM
HPU extension package for vLLM
A high-throughput and memory-efficient inference and serving engine for LLMs
Iterable-based offline generation helpers for vLLM.
vLLM Kunlun3 backend plugin
A collection of useful util functions