Agent frameworks & infrastructure

Model serving & inference

775 agents · most starred first

Model serving & inference, page 7 of 8

vllm-manager

Multi-instance vLLM cluster orchestration and log management

vllm-marconi-offload

Two-tier (RAM + SSD) KV cache offload connector for vLLM with Marconi-style reuse-aware eviction.

vllm-marenostrum

Super simple vLLM server launcher for SLURM/HPC with nested config support

vllm-mbart
vllm-messages

This name has been reserved using Reserver

vllm-mindspore

A high-throughput and memory-efficient inference and serving engine for LLMs

vllm-mini

vLLM mini.

vllm-mock

Provide mock instance to test vllm without CUDA or any GPUs.

vllm-musa

vLLM platform plugin for Moore Threads MUSA GPUs

vllm-nccl-cu11
vllm-nccl-cu12
vllm-npu

A high-throughput and memory-efficient inference and serving engine for LLMs

vllm-omni

A framework for efficient model inference with omni-modality models

vllm-online

A high-throughput and memory-efficient inference and serving engine for LLMs

vllm-playground

A web interface for managing and interacting with vLLM servers

vllm-plugin-meralion2

A vLLM plugin to register the MERaLiON-2-10B model architecture with vLLM’s plugin system.

vllm-rbln

vLLM plugin for RBLN NPU

vllm-recorder
vllm-responses

This name has been reserved using Reserver

vllm-rocm

A high-throughput and memory-efficient inference and serving engine for LLMs with AMD GPU support

vllm-rs

A minimal, high-performance large language model (LLM) inference engine implementing vLLM in Rust.

vllm-sdk

Minimal Python SDK for the vLLM API

vllm-speculative-autoconfig

Automatic configuration planner for vLLM - Eliminate the guesswork of configuring vLLM by automatically determining optimal parameters

vllm-spyre

vLLM plugin for Spyre hardware support

vllm-spyre-next

Next iteration of vllm-spyre on the torch-spyre stack

vllm-swift

vLLM Metal plugin powered by mlx-swift — high-performance LLM inference on Apple Silicon

vllm-test-tpu

A high-throughput and memory-efficient inference and serving engine for LLMs

vllm-tgis-adapter

vLLM adapter for a TGIS-compatible grpc server

vllm-tpu

A high-throughput and memory-efficient inference and serving engine for LLMs

vllm-tuner

A Python package for tuning vLLM hyperparameters.

vllm-usf

vLLM-USF: A high-throughput and memory-efficient inference engine for LLMs (USF Custom Build)

vllm-wizard

CLI tool for vLLM configuration generation and GPU sizing

vllm-xft

A high-throughput and memory-efficient inference and serving engine for LLMs

vllm-xpu
vllm-xpu-kernels

A high-throughput and memory-efficient inference and serving engine for LLMs

vllmctl

CLI tool for launching and managing vllm model servers via SSH and tmux

vllmlp

Collections of Logits Processors for VLLM

watsonx-agent-deploy

Deploy multiple AI agents to watsonx.ai at once

wicky-vllm

a lightweight vLLM implementation built from scratch

zllm-kit

LLM quantization framework

zllm-lib

LLM quantization framework

zllm-zse

ZSE - Z Server Engine: Ultra memory-efficient LLM inference engine

eidolon-ai

Eidolon AI review: compare features, pricing, use cases, access model, and alternatives for this Model Serving agent in 2026. AI Agent Server for the Enterprise

fatcat-gpt

胖貓專屬嘴臭LLM推理API

vllmpytop

A btop-style terminal UI for monitoring a vLLM instance and its GPU in real time.

azure-ai-agentserver-optimization

Optimization config loader for Azure AI Hosted Agents

llm-engine-kitty

A lightweight LLM inference engine supporting OpenAI-compatible APIs.

apohara-vllm-plugin

Apohara ContextForge plugin for vLLM V1 — multi-agent KV-cache coordination, JCR Safety Gate (INV-15), RotateKV INT4 hooks, on AMD Instinct MI300X.

langchain-zerogpu

LangChain tools for ZeroGPU: A compute-efficient inference provider for apps and agents — purpose-built small and nano language models on an edge network that run the repeatable tasks frontier models shouldn’t, ~10x faster and 50%+ cheaper. Auto-scaling, with zero GPU infrastructure. Plug in and you’re live.

open-vllm-sdk

Metapackage redirect: Please use open-vllm instead.

semblend-vllm-connector

Out-of-tree vLLM KVConnector for SemBlend semantic KV donor discovery

vllm-grpc-client

Lean async gRPC client for the vllm-grpc frontend

vllm-grpc-frontend

gRPC frontend server exposing vLLM's V1 engine over protobuf/gRPC

vllm-grpc-gen

Generated protobuf/gRPC stubs for the vllm-grpc frontend, proxy, and client

vllm-grpc-proxy

REST-to-gRPC proxy server for the vllm-grpc frontend

byted-xllm-torch-inferencer

This is a reserved placeholder package to mitigate dependency confusion risks.

jina-v4-vllm-plugin

vLLM out-of-tree model: Jina Embeddings v4 multi-vector (token_embed) on Qwen2.5-VL

vllm-cpu-nightly

A high-throughput and memory-efficient inference and serving engine for LLMs

llm-jp-vllm

vLLM plugins for LLM-jp models

llmpu

Large Language Model Processing Unit

vllmstat

nvtop for vLLM — an interactive terminal dashboard for vLLM serving performance and GPU/fleet monitoring

llm-batch-inference

LLM Batch Inference

pace-vllm

PACE platform plugin for vLLM CPU inference on AMD EPYC processors.

vllm-mblt

An out-of-tree vLLM plugin for Mobilint NPU runtime integration.

open-vllm

The official library for vLLM client network and preflight validation engine.

vllm-neuron
raif-vllm

One vLLM plugin for transparent RAIF token savings — install it and existing OpenAI clients get RAIF on tools & response_format with no proxy and no client changes.

condi-llm

Distributed training and serving framework for models up to 405B parameters.

@sap-ai-sdk/llm-batch

> [!warning] > This package is still in **beta** and is subject to breaking changes. Use it with caution.

emmy-llm

Deploy and benchmark LLM inference on GPU servers using vLLM

lfx-vllm

vLLM as a first-class Langflow model provider, packaged as a standalone Extension Bundle.

llm-engine-rs

A high-performance LLM inference engine in Rust

vllmops

Declarative control plane and TUI for bare-metal vLLM servers.

@intentsolutionsio/jeremy-vertex-engine

Vertex AI Agent Engine deployment inspector and runtime validator

inferlab-integration-vllm

Inferlab adapter integration for vLLM

@inferencesh/sdk

Official JavaScript/TypeScript SDK for inference.sh - Run AI models with a simple API

LLMmodelHub

Turn any machine or Google Colab notebook into a one-line AI model server with an auto-generated public API.

amlllm-openai-server

Open AI API compatiable LLM server for Amlogic Device

aws_sdk_bedrock

Python client for Amazon Bedrock

evoke-vllm

Relevance-driven CPU KV-cache offload policy for stock vLLM

fastllm-turbo

Mixed-precision LLM acceleration: faster with same output

friday-framework-llm

LLM inference layer for Friday Framework

hotwire-vllm

CUDA-graph-safe activation steering plugin for vLLM

inferlab-integration-tensorrt-llm

Inferlab adapter integration for TensorRT-LLM

invarlock-runtime-tensorrt-llm

First-party TensorRT-LLM runtime provider add-in for InvarLock

langchain-commoncompute

LangChain provider for Common Compute — plug Apple-Silicon-powered LLMs and embeddings into your LangChain app.

llama-dyno

Auto-tune and benchmark llama.cpp / ik_llama.cpp inference on NVIDIA, AMD (ROCm), and Apple Silicon GPUs

llama-index-commoncompute

LlamaIndex integration for Common Compute — Apple-Silicon-powered LLMs and embeddings for your LlamaIndex pipeline.

llm-dequant

Dequantize compressed-tensors NVFP4 LLM checkpoints to dense safetensors (streaming, constant memory)

llm-flops

Accurate FLOPs, memory, latency, and energy estimation for Large Language Models.

memkv-vllm

MemKV backend for vLLM native KV offloading: a secondary tier behind the CPU pool (TieringOffloadingSpec) or MemKV as the offload medium itself (MemKVOffloadingSpec).

openai-http

OpenAI v1 API-compatible HTTP service backed by HuggingFace Transformers

openlake-vllm

OpenLake high performance KV cache client and server for vLLM.

snowllm

Hand-written HIP kernels for LLM inference on AMD Ryzen AI Max (Strix Halo)

snowllm-kernels

Compiled HIP kernels for SnowLLM on AMD Ryzen AI Max (Strix Halo)

vllm-bnb-plugin

Out-of-tree bitsandbytes quantization plugin for vLLM

vllm-cleaner

A utility to automatically kill vLLM processes and release GPU memory.

vllm-omni-humming

Companion plugin: load & serve pre-quantized humming Qwen-Image DiT pipelines on stock vLLM-Omni

vllm-omni-neuron
@looprun-ai/server

looprun model server: expose governed LoopRunAgents behind an OpenAI-compatible /v1/chat/completions endpoint, so any harness that speaks the OpenAI protocol can call a governed agent as if it were a model. The full governed turn (guards, tools, redrive)

More in Agent frameworks & infrastructure