Agent frameworks & infrastructure

Model serving & inference

775 agents · most starred first

Model serving & inference, page 6 of 8

open-llm-server
openai-deploy
openllm

OpenLLM: Self-hosting LLMs Made Easy.

openllm-client

OpenLLM Client: Interacting with OpenLLM HTTP/gRPC server, or any BentoML server.

openllm-core

OpenLLM Core: Core components for OpenLLM.

openvino-openai-api

OpenAI Compatible API Server using OpenVINO GenAI

osc-llm

轻量级大模型推理工具,专注于模型推理延迟,注重框架易用性和可拓展性。

ovllm

One-line vLLM wrapper with gorgeous DSPy integration

paragon-py

Paragon AI — Python bindings (ctypes) with embedded cross-vendor GPU backends

pegaflow-llm

High-performance key-value storage engine with Python bindings

pegaflow-llm-cu13

High-performance key-value storage engine with Python bindings

plllm-mlx

Standalone MLX-based LLM inference service with OpenAI compatible API

pnap-network-storage-api

Network Storage API

polarisllm

🌟 The Ultimate Multi-Model LLM Runtime Platform - Deploy, manage, and serve 300+ language models with OpenAI-compatible APIs. Built on ms-swift for production-ready performance.

prime-vllm

Best collective comms library in the solar system

prismllm

PrismLLM: Hardware-agnostic LLM inference library

processfork-vllm

ProcessFork plugin for vLLM ≥0.10 — paged-KV-cache snapshot/restore via batch-invariant kernels.

protollm-api
protollm-worker
pylibllm

libLLM is an open-source project tailored for efficient inference of large language models (LLMs).

pyllama

🦙 LLaMA: Open and Efficient Foundation Language Models in A Single GPU

pyllama-server

Python wrapper for llama.cpp server - OpenAI API compatible backend with auto device detection

qllm

A general x-bit quantization engine for LLMs,[2-8] bits, awq/gptq/hqq/vptq

qwerky-vllm-models

vLLM plugin for Qwerky AI MambaInLlama hybrid models

rungpt

An open-source cloud-native of large multi-modal models (LMMs) serving framework.

scale-llm-engine

Scale LLM Engine Python client

scalellm

A high-performance inference system for large language models.

seeta-dragon

Dragon: A Computation Graph Virtual Machine Based Deep Learning Framework

sentry-fs-node-storage

Sentry File System Node Storage

sentry-riak-node-storage

Sentry Riak Node Storage

serverless-llm
serverless-llm-cli

A simple description of your package

serverless-llm-serve

A simple description of your package

serverless-llm-store
serving_agent

A middleware for model serving to speedup online inference.

shard-llms

Shard Large Language Models

shopee-aip-rayllmbatch

A small example package

sinapsis-vllm

Sinapsis templates for text, batch, streaming, and multimodal generation with vLLM.

smol-vllm

From-scratch paged-attention inference engine: paged KV cache, continuous batching, preemption

sovereign-pulse-vllm

Sovereign Pulse physics gate — vLLM LogitsProcessor plugin

squeezellm

squeezellm

strands-vllm

Community vLLM provider utilities for Strands Agents (OpenAI-compatible).

stressllm

Find the breaking point of your local LLM hardware.

substrai-lambdallm

Serverless-native LLM orchestration framework for AWS Lambda

superduper-vllm

Superduper allows users to work with self-hosted LLM models via [vLLM](https://github.com/vllm-project/vllm).

svllm

serve your llm

sych-llm-playground

Sych LLM Playground is a command line tool deploy and interact with language models on the cloud.

telellm

A high-throughput and memory-efficient inference and serving engine for LLMs

tensorrt-llm

TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs.

thaw-vllm

The fork primitive for LLM inference. Snapshot a running session — weights + KV cache + scheduler state — and hydrate it into N divergent children that skip prefill. For RL rollouts, parallel coding agents, agent branching. Supports vLLM and SGLang.

tinyedgellm

A modular framework for LLM quantization, structured pruning, and edge deployment

tokenspeed-trtllm-attn

Name reserved for the tokenspeed-trtllm-attn project.

tokenspeed-trtllm-common

Name reserved for the tokenspeed-trtllm-common project.

tokenspeed-trtllm-gemm

Name reserved for the tokenspeed-trtllm-gemm project.

tokenspeed-trtllm-kernel

Standalone TensorRT-LLM CUDA kernels as PyTorch custom ops

tokenspeed-trtllm-moe

Name reserved for the tokenspeed-trtllm-moe project.

torch-llm-kernels

CUDA Kernels for Large Language Models (LLMs), compatible with torch.compile.

torch_to_nnef_llm

LLM and PEFT export to NNEF via torch-to-nnef.

transformers-openai-api

An OpenAI Completions API compatible server for NLP transformers models

tritonllm

Flexible and modular LLM inference for mini-batch

trtllm-kernel

Name reserved for the trtllm-kernel project.

turboagents

Independent TurboQuant-style compression infrastructure for KV cache and RAG.

turboquant-llm

Reference implementation of TurboQuant-style vector/KV quantization (rotation + scalar quant + QJL residual).

turboquant-plus-vllm

TurboQuant+ compression for vLLM. 4.3x weight compression + 3.7x KV cache, zero calibration.

turboquant-vllm

TurboQuant KV cache compression for vLLM — fused Triton kernels, 3.76x compression, 3.7x faster decode on RTX 4090

types-boto3-bedrock-runtime

Type annotations for boto3 BedrockRuntime 1.44.0 service generated with mypy-boto3-builder 8.12.0

ultravox-vllm
unreasonable-llama

HTTP API bindings for llama.cpp built-in example server

vineyard-llm

Vineyard llm kv cache

vital-llm-cluster-mgr

Vital LLM Cluster Manager

vlllm

A utility package for text generation using vLLM with multiprocessing support.

vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

vllm-acc

A high-throughput and memory-efficient inference and serving engine for LLMs

vllm-allocator-adaptor

vLLM Allocator Adaptor (C/CUDA/Python) using callback shims

vllm-ascend

vLLM Ascend backend plugin

vllm-block

Block implementation of vLLM

vllm-bootstrap-client

gRPC client for vllm-bootstrap inference service

vllm-canon

vLLM plugin: out-of-tree registration of canon-layer architectures (e.g. LlamaCanonForCausalLM from PhysicsLM4)

vllm-cli

A CLI tool to conveniently serve LLMs with vLLM

vllm-client

Client for the vLLM API with minimal dependencies

vllm-cluster-manager

Deploy, manage, and monitor vLLM instances across a GPU cluster from a single web dashboard.

vllm-consul

A high-throughput and memory-efficient inference and serving engine for LLMs

vllm-cpm

A high-throughput and memory-efficient inference and serving engine for LLMs

vllm-cpu

A high-throughput and memory-efficient inference and serving engine for LLMs

vllm-cpu-amxbf16

vLLM CPU inference engine (AVX512 + VNNI + BF16 + AMX optimized)

vllm-cpu-avx512

vLLM CPU inference engine (AVX512 optimized)

vllm-cpu-avx512bf16

vLLM CPU inference engine (AVX512 + VNNI + BF16 optimized)

vllm-cpu-avx512vnni

vLLM CPU inference engine (AVX512 + VNNI optimized)

vllm-df11

Dfloat11 plugin for vLLM

vllm-dolphin
vllm-emissary

A high-throughput and memory-efficient inference and serving engine for LLMs

vllm-factory

The LEGO set for custom vLLM model plugins — build, test, and deploy custom encoders, poolers, and kernels

vllm-fixed

A high-throughput and memory-efficient inference and serving engine for LLMs

vllm-flash-attn

Forward-only flash-attn

vllm-gguf-plugin

Out-of-tree GGUF quantization plugin for vLLM

vllm-hpu-extension

HPU extension package for vLLM

vllm-hust

A high-throughput and memory-efficient inference and serving engine for LLMs

vllm-iter

Iterable-based offline generation helpers for vLLM.

vllm-kunlun

vLLM Kunlun3 backend plugin

vllm-logits

A collection of useful util functions

More in Agent frameworks & infrastructure