Agent frameworks & infrastructure
Model serving & inference
775 agents · most starred first
Model serving & inference, page 7 of 8
Multi-instance vLLM cluster orchestration and log management
Two-tier (RAM + SSD) KV cache offload connector for vLLM with Marconi-style reuse-aware eviction.
Super simple vLLM server launcher for SLURM/HPC with nested config support
This name has been reserved using Reserver
A high-throughput and memory-efficient inference and serving engine for LLMs
vLLM mini.
Provide mock instance to test vllm without CUDA or any GPUs.
vLLM platform plugin for Moore Threads MUSA GPUs
A high-throughput and memory-efficient inference and serving engine for LLMs
A framework for efficient model inference with omni-modality models
A high-throughput and memory-efficient inference and serving engine for LLMs
A web interface for managing and interacting with vLLM servers
A vLLM plugin to register the MERaLiON-2-10B model architecture with vLLM’s plugin system.
vLLM plugin for RBLN NPU
This name has been reserved using Reserver
A high-throughput and memory-efficient inference and serving engine for LLMs with AMD GPU support
A minimal, high-performance large language model (LLM) inference engine implementing vLLM in Rust.
Minimal Python SDK for the vLLM API
Automatic configuration planner for vLLM - Eliminate the guesswork of configuring vLLM by automatically determining optimal parameters
vLLM plugin for Spyre hardware support
Next iteration of vllm-spyre on the torch-spyre stack
vLLM Metal plugin powered by mlx-swift — high-performance LLM inference on Apple Silicon
A high-throughput and memory-efficient inference and serving engine for LLMs
vLLM adapter for a TGIS-compatible grpc server
A high-throughput and memory-efficient inference and serving engine for LLMs
A Python package for tuning vLLM hyperparameters.
vLLM-USF: A high-throughput and memory-efficient inference engine for LLMs (USF Custom Build)
CLI tool for vLLM configuration generation and GPU sizing
A high-throughput and memory-efficient inference and serving engine for LLMs
A high-throughput and memory-efficient inference and serving engine for LLMs
CLI tool for launching and managing vllm model servers via SSH and tmux
Collections of Logits Processors for VLLM
Deploy multiple AI agents to watsonx.ai at once
a lightweight vLLM implementation built from scratch
LLM quantization framework
LLM quantization framework
ZSE - Z Server Engine: Ultra memory-efficient LLM inference engine
Eidolon AI review: compare features, pricing, use cases, access model, and alternatives for this Model Serving agent in 2026. AI Agent Server for the Enterprise
胖貓專屬嘴臭LLM推理API
A btop-style terminal UI for monitoring a vLLM instance and its GPU in real time.
Optimization config loader for Azure AI Hosted Agents
A lightweight LLM inference engine supporting OpenAI-compatible APIs.
Apohara ContextForge plugin for vLLM V1 — multi-agent KV-cache coordination, JCR Safety Gate (INV-15), RotateKV INT4 hooks, on AMD Instinct MI300X.
LangChain tools for ZeroGPU: A compute-efficient inference provider for apps and agents — purpose-built small and nano language models on an edge network that run the repeatable tasks frontier models shouldn’t, ~10x faster and 50%+ cheaper. Auto-scaling, with zero GPU infrastructure. Plug in and you’re live.
Metapackage redirect: Please use open-vllm instead.
Out-of-tree vLLM KVConnector for SemBlend semantic KV donor discovery
Lean async gRPC client for the vllm-grpc frontend
gRPC frontend server exposing vLLM's V1 engine over protobuf/gRPC
Generated protobuf/gRPC stubs for the vllm-grpc frontend, proxy, and client
REST-to-gRPC proxy server for the vllm-grpc frontend
This is a reserved placeholder package to mitigate dependency confusion risks.
vLLM out-of-tree model: Jina Embeddings v4 multi-vector (token_embed) on Qwen2.5-VL
A high-throughput and memory-efficient inference and serving engine for LLMs
vLLM plugins for LLM-jp models
Large Language Model Processing Unit
nvtop for vLLM — an interactive terminal dashboard for vLLM serving performance and GPU/fleet monitoring
LLM Batch Inference
PACE platform plugin for vLLM CPU inference on AMD EPYC processors.
An out-of-tree vLLM plugin for Mobilint NPU runtime integration.
The official library for vLLM client network and preflight validation engine.
One vLLM plugin for transparent RAIF token savings — install it and existing OpenAI clients get RAIF on tools & response_format with no proxy and no client changes.
Distributed training and serving framework for models up to 405B parameters.
> [!warning] > This package is still in **beta** and is subject to breaking changes. Use it with caution.
Deploy and benchmark LLM inference on GPU servers using vLLM
vLLM as a first-class Langflow model provider, packaged as a standalone Extension Bundle.
A high-performance LLM inference engine in Rust
Declarative control plane and TUI for bare-metal vLLM servers.
Vertex AI Agent Engine deployment inspector and runtime validator
Inferlab adapter integration for vLLM
Official JavaScript/TypeScript SDK for inference.sh - Run AI models with a simple API
Turn any machine or Google Colab notebook into a one-line AI model server with an auto-generated public API.
Open AI API compatiable LLM server for Amlogic Device
Python client for Amazon Bedrock
Relevance-driven CPU KV-cache offload policy for stock vLLM
Mixed-precision LLM acceleration: faster with same output
LLM inference layer for Friday Framework
CUDA-graph-safe activation steering plugin for vLLM
Inferlab adapter integration for TensorRT-LLM
First-party TensorRT-LLM runtime provider add-in for InvarLock
LangChain provider for Common Compute — plug Apple-Silicon-powered LLMs and embeddings into your LangChain app.
Auto-tune and benchmark llama.cpp / ik_llama.cpp inference on NVIDIA, AMD (ROCm), and Apple Silicon GPUs
LlamaIndex integration for Common Compute — Apple-Silicon-powered LLMs and embeddings for your LlamaIndex pipeline.
Dequantize compressed-tensors NVFP4 LLM checkpoints to dense safetensors (streaming, constant memory)
Accurate FLOPs, memory, latency, and energy estimation for Large Language Models.
MemKV backend for vLLM native KV offloading: a secondary tier behind the CPU pool (TieringOffloadingSpec) or MemKV as the offload medium itself (MemKVOffloadingSpec).
OpenAI v1 API-compatible HTTP service backed by HuggingFace Transformers
OpenLake high performance KV cache client and server for vLLM.
Hand-written HIP kernels for LLM inference on AMD Ryzen AI Max (Strix Halo)
Compiled HIP kernels for SnowLLM on AMD Ryzen AI Max (Strix Halo)
Out-of-tree bitsandbytes quantization plugin for vLLM
A utility to automatically kill vLLM processes and release GPU memory.
Companion plugin: load & serve pre-quantized humming Qwen-Image DiT pipelines on stock vLLM-Omni
looprun model server: expose governed LoopRunAgents behind an OpenAI-compatible /v1/chat/completions endpoint, so any harness that speaks the OpenAI protocol can call a governed agent as if it were a model. The full governed turn (guards, tools, redrive)