Agent frameworks & infrastructure

Model serving & inference

775 agents · most starred first

Model serving & inference, page 8 of 8

dragongui

A Python application toolkit for GPU-native data tools.

vllm-kvblockd

vLLM native KV-transfer adapter backed by kvblockd (KVConnectorBase_V1 + SecondaryTierManager)

otf-llm

High-performance hybrid LLM inference engine with Adaptive Non-Uniform 2-Bit Quantization (Lloyd-Max + Fused Triton INT2 GEMM), RLM Context-as-a-Variable engine, 98.2% logit parity, and Zero-RAM quantizer.

capo-bedrock-runtime

Typed, async Python SDK for AWS Bedrock Runtime.

exionos-vllm

vLLM architecture plugin for Exionos X1

needle-openai

OpenAI-compatible HTTP API for the Needle 2 tool-calling model

oragon

Local-first LLM serving for software that has to work: a resident engine, typed errors, validated extraction and an OpenAI-compatible API — on hardware you don't control, with data that can't leave.

pumpwood-deploy-model-llm

Package for deploy LLM Model Service

distributed-llm

Distributed LLM inference — pool GPUs across multiple devices to run models no single machine can handle

hetzner-haystack

An integration between the Hetzner Inference API and Haystack

vllm-beam-search-plugin

Beam search plugin for vLLM v1 with CoW block table forking

llm-d-planner

Confidently navigate LLM deployments from concept to production

pytllm

TLLM runs 70B large language models on a single 4GB GPU without quantization, distillation or pruning. 405B Llama 3.1 on 8GB, DeepSeek-V3 671B on ~12GB, Kimi K3 2.8T on under 4GB, Qwen3.8-27B on 3.3GB.

fitcheck-llm

Estimate LLM training and serving memory from model configs; measured training evidence covers one Tesla T4.

fortel-agent

Run Fortel machine-learning jobs on your own machine.

streamllm

Run bigger LLMs on smaller GPUs through intelligent, asynchronous layer streaming.

vllm-hust-clm-lifecycle

CLM: context-lifecycle management perception plugin for vLLM-HUST (request-lifecycle event sink)

vllm-hust-dla

DLA: decode-length-aware preemption & admission for vLLM-HUST

vllm-optimizer

Benchmark-driven configuration optimization for vLLM inference servers

vllm-sbw

SBW watermarking logits processor for vLLM

vllm-spark2-5-plugin

Spark2_5 support for vLLM, maintained out of tree

cloudmesh-ai-llm

Extension vllm created by CME

flyteplugins-llamacpp

llama.cpp plugin for flyte

microgen-llm

Lightweight, modular, hardware-aware LLM inference engine & empirical research testbed.

vllm-hi

Low-downtime hot migration for vLLM inference services, via monkey-patching and a plugin (no vLLM source changes).

vllm-hust-vspec

Out-of-tree speculative decoding plugin for vLLM-HUST on Ascend

wllm-server

A vLLM-style, production-grade LLM inference server built natively for Windows (single GPU, single machine)

Geodd

Geodd provides fast, steady AI model hosting through serverless inference and dedicated GPUs. It supports OpenAI-compatible tools, regional deployment, usage tr

awswarm

Sub-layer fragment placement for GPUs too small to hold one full transformer layer -- capacity-proportional splitting, column/row-parallel execution, verified against an unsplit reference.

axm-vllm

Package name reserved — implementation coming soon

duplexjev-vllm

vLLM plugin for DuplexJev speech-decision models (Qwen3-ASR encoder + Ultravox-style projector).

fireworks2bedrock-migrate

One-shot migration of Kimi K3 inference from Fireworks to Amazon Bedrock for Python repositories

llm2jev

Any chat model, any engine (SGLang, vLLM, transformers, MLX) as a Jev-compatible probability decision service

openai-maple

OpenAI-compatible HTTP API for the deepgrove/maple-preview reasoning model

typellm

Type-Safe Decoding for Autoregressive LLMs

vllm-ascend-kvcompress-hust

TriAttention KV-cache compression plugin for vLLM Ascend

vllm-ascend-quant-ext

Fail-closed runtime support for versioned Ascend quantized model artifacts

vllm-ascend-quantized-kv-cache

Runtime-loadable INT8 KV-cache attention backend for vLLM-Ascend-HUST

vllm-ascend-split-batch

vLLM-HUST plugin carriers for vllm-ascend: decode FIA de-mask, RoPE variant fixes, zero-cost wiring, MLP chunking and inert split-batch planning

vllm-betterscale

Modular graph and replay optimizations for vLLM Ascend

vllm-dgxspark-psychoscales

A high-throughput and memory-efficient inference and serving engine for LLMs

vllm-hust-bidkv

BidKV — CompressionBid protocol layer for KV cache scheduling primitives

vllm-hust-operator-optimizations

Migration package for OPset, the vLLM-HUST operator optimization bundle

vllm-hust-opset

Pluggable kernel replacements and operator fusions for the vLLM-HUST Ascend backend

vllm-ipc-cache

Zero-copy CUDA/MACA IPC weight pre-load cache for vLLM instant restart

@onivoro/server-agentic-bedrock

AWS Bedrock streaming model provider for @onivoro/server-agentic, with pluggable per-model wire protocols

iaexplore-llm-profiler

Deterministic high-precision VRAM, GQA KV-Cache, MoE Memory Bandwidth, and llama.cpp layer offloading profiler for 2026 AI models and silicon.

kern-llm

The kern runtime for Python, and the plugins that run a kern manifest as a vLLM or SGLang model

langchain-flexai

An integration package connecting FlexAI and LangChain

llm-kernel

Tensor math, machine learning, deep learning and model inference helpers: reusable functions and best practices.

llm-sizer

LLM VRAM calculator and cheapest AWS GPU recommender for vLLM, DeepSeek-R1, Llama 70B, and Qwen. JSON output for AI agents.

mimo-asr-vllm

Fast MiMo-V2.5-ASR inference and OpenAI-compatible serving with vLLM

pion-vllm-mlx

Pion prompt cache for mlx-lm on Apple Silicon — reuse a prefix's K/V across processes and restarts

ruvector-ruvllm

Bounded native Rust key value cache and attention primitives for Python LLM pipelines

sabi-mla-vllm

vLLM plugin that serves sabi-noema-mla (shared pre-RoPE rotary MLA, rank 512, on the 16 full-attention layers of a Qwen3.5/3.8 hybrid)

servellm

Expose a local LLM runtime as a reliable HTTP/WebSocket service.

tensorrt-edgellm

TensorRT Edge-LLM model build and inference package

vectorch-sllm

sllm: a high-performance LLM inference engine (placeholder; first release coming soon)

velocityllm

SLA-aware request scheduling for LLM serving, with a mock engine, an API server and benchmark tools

vllm-iquest-q1

IQuest-Q1 models and parsers for vLLM

vllm-jax
vllm-jax-core
vllm-service-ple

vLLM plugin: keeps the Flash-Next PLE table out of the vLLM process (per-machine agent with a shared cache, local file and/or central server)

vllm-torchtpu

A high-throughput and memory-efficient inference and serving engine for LLMs

vllm-torchtpu-core
vllm-xtu-moe

XTU (X Transformers Unity) — a mainline-vLLM hybrid MoE plugin (CPU experts + GPU attention/long prefill). Bundles the xiaotu CPU MoE engine (AVX512-VNNI/BF16: MXFP4/FP8/INT4/BF16) so no-AMX x86 hosts (AMD EPYC) get usable CPU MoE backends through FusedMoEFactory.

volcenginesdk-storageebs

Volcengine storageebs SDK for Python

anet-kllm

Pull, run and seal LLMs and diffusion models on SGLang and vLLM.

llm-gpu-cost-calculator

CLI tool to audit real cloud GPU TCO, spot interruption penalty, and generative multimodal AI costs.

vllm-quantpack

vLLM plugin: serve EXL3 (exllamav3 trellis) checkpoints and the model-family fixes they need, with no patch to vLLM

domyn-swarm

CLI and programmatic API to launch LLM inference endpoints on Slurm, using simple configuration files

laidline-vllm

vLLM Model Runner V2 logits processor for the text watermark plus an ack-header middleware.

telys-vllm

Telys external KV/prefix-cache connector for vLLM (KVConnectorBase_V1)

vllm-metis

vLLM plugin for Lernex's Metis-1.6 models: the Metis16ForCausalLM architecture plus the metis reasoning and tool-call parsers

colabllm

Run model services on Google Colab with managed tunnels and authentication

More in Agent frameworks & infrastructure