Agent frameworks & infrastructure

Model serving & inference

Model serving and inference projects run and serve trained models rather than build them: vLLM provides a high-throughput, memory-efficient engine for LLM inference, while gpustack manages GPU clusters and exposes vLLM or SGLang backends as on-demand instances. They differ in the deployment concern each handles, from quantization and batching to OpenAI-compatible endpoints and cluster scheduling.

775 agents · most starred first

Top 100 of 775

vllm★ 97,261

A high-throughput and memory-efficient inference and serving engine for LLMs

ColossalAI★ 41,420

Making large AI models cheaper, faster and more accessible

FastChat★ 39,527

An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.

kubeflow★ 15,973

The Cloud Native AI Platform

OpenLLM★ 12,608

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

gpustack★ 6,090

A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.

mesh-llm★ 4,179

Distributed AI/LLM for the people. Share compute privately or publicly to power your agents and chat.

Rapid-MLX★ 4,166

Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. Up to 4× faster than Apple's MLX (mlx-lm) on the same weights.

csghub★ 4,083

CSGHub is a brand-new open-source platform for managing LLMs, developed by the OpenCSG team. It offers both open-source and on-premise/SaaS solutions, with features comparable to Hugging Face. Gain full control over the lifecycle of LLMs, datasets, and agents, with Python SDK compatibility with Hugging Face. Join us! ⭐️

FedML★ 4,068

FEDML - The unified and scalable ML library for large-scale distributed training, model serving, and federated learning. FEDML Launch, a cross-cloud scheduler, further enables running any AI jobs on any GPU cloud or on-premise cluster. Built on this library, TensorOpera AI (https://TensorOpera.ai) is your generative AI platform at scale.

FastDeploy★ 3,722

High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle

sie★ 3,469

Open-source inference server and production cluster for all the models your agent needs.

TensorFold★ 2,352

LLM Inference Engine for Metal, CUDA and Vulkan.

intel-extension-for-transformers★ 2,172

⚡ Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for LLMs; run LLMs efficiently on Intel Platforms⚡

dstack★ 2,170

Vendor-agnostic orchestration for training, inference and agentic workloads across NVIDIA, AMD, TPU, and Tenstorrent on clouds, Kubernetes, and bare metal.

vllm-mlx★ 1,656

High-performance OpenAI and Anthropic compatible LLM inference server for Apple Silicon. Native MLX, continuous batching, multimodal models, MCP tool calling, and Claude Code support.

langchain-serve★ 1,629

⚡ Langchain apps in production using Jina & FastAPI

parallax★ 1,429

Parallax is a distributed model serving framework that lets you build your own AI cluster anywhere

Jlama★ 1,310

Jlama is a modern LLM inference engine for Java

langcorn★ 940

⛓️ Serving LangChain LLM apps and agents automagically with FastApi. LLMops

llamafarm★ 837

Deploy any AI model, agent, database, RAG, and pipeline locally or remotely in minutes

C2C★ 782

[ICLR'26] The official code implementation for "Cache-to-Cache: Direct Semantic Communication Between Large Language Models"

TensorSharp★ 681

A native .NET LLM inference engine and agent runtime for GGUF models. TensorSharp provides a console application, a web-based chatbot interface, iPhone App, and Ollama/OpenAI-compatible HTTP APIs for programmatic access. It supports Windows/MacOS/iOS/Linux with full GPU capability

attorch★ 607

A subset of PyTorch's neural network modules, written in Python using OpenAI's Triton.

OpenArc★ 551

Inference engine for Intel devices. Serve LLMs, VLMs, Whisper, Kokoro-TTS, Embedding and Rerank models over OpenAI endpoints.

Flide★ 547

🔥🔥 The extremely fast inference engine for decision model

KVarN★ 521

KVarN is a native vLLM KV-cache quantization backend for your agents: 3-5x more context, throughput above FP16, and FP16-level accuracy. Calibration-free, one flag.

vllm-cli★ 507

A command-line interface tool for serving LLM using vLLM.

ThunderAgent★ 485

A simple, fast and robust program-aware agentic inference system.

super-json-mode★ 398

Low latency JSON generation using LLMs ⚡️

AKO4ALL★ 384

Agentic Kernel Optimization for All — automated GPU kernel optimization for any kernel, any hardware, any language

Bittune★ 361

Local-first engineering agent that deploys, benchmarks and tunes vLLM/SGLang inference services on your own GPUs, with auditable run evidence.

chat.petals.dev★ 319

💬 Chatbot web app + HTTP and Websocket endpoints for LLM inference with the Petals client

auto-gpu-kernel★ 319

Winner 🏆 (Agent-only) MLSys 2026 - FlashInfer AI Kernel Generation Contest for the DeepSeek Sparse Attention (DSA) track with an average speedup of 34.93x

flama★ 306

The production framework for Predictive and Generative AI. Serve any model as an API in one line, with OpenAI/Anthropic/Ollama-compatible endpoints, a built-in chat UI, and native MCP.

TensorRT-Model-Connect★ 297

From PyTorch model to end-to-end TensorRT inference experience in two commands—AI-native, cross-platform, and built for the best possible user experience.

Hyperloom★ 282

An agentic system that auto-optimizes LLM workloads on AMD GPUs.

gpt_server★ 254

gpt_server是一个用于生产级部署LLMs、Embedding、Reranker、ASR、TTS、文生图、图片编辑和文生视频的开源框架。

dgx-spark-qwen38★ 244

dgx-spark-qwen38 is an open-source installer that turns an NVIDIA DGX Spark, or any GB10 machine (ASUS, Dell, HP, Lenovo, MSI, GIGABYTE, Acer), into a private AI server in one command: measured model lanes, OpenAI and Anthropic APIs, and a local cockpit.

openai_trtllm★ 221

OpenAI compatible API for TensorRT LLM triton backend

llama-api-server★ 207

A OpenAI API compatible REST server for llama.

gpt-j★ 206

A GPT-J API to use with python3 to generate text, blogs, code, and more

AutoMegaKernel★ 166

An agent harness that compiles a model into one provably-correct, self-retargeting CUDA megakernel and self-tunes it past cuBLAS at batch-1 LLM decode, paper: https://arxiv.org/abs/2606.09682

grps_trtllm★ 160

Higher performance OpenAI LLM service than vLLM serve: A pure C++ high-performance OpenAI LLM service implemented with GPRS+TensorRT-LLM+Tokenizers.cpp, supporting chat and function call, AI agents, distributed multi-GPU inference, multimodal capabilities, and a Gradio chat interface.

ialacol★ 147

🪶 Lightweight OpenAI drop-in replacement for Kubernetes

sndr_core_engine★ 133

SNDR Core Engine (Genesis) — vLLM runtime patch-overlay for Qwen3.6 + Gemma4 on consumer NVIDIA (Ampere sm_86, 2× A5000/3090). Qwen3.6-35B-A3B FP8 ~240 tok/s, 27B-int4 hybrid GDN+Mamba, Gemma4 26B/31B AWQ, 256K ctx. 321 patches: TurboQuant k8v4 KV, MTP/DFlash spec-decode, FULL cudagraph, hybrid GDN. vLLM pin dev424 + Control Center GUI.

llm-inference★ 124

Large Language Model (LLM) Inference API and Chatbot

SLED★ 124

SLED: Self Logits Evolution Decoding for Improving Factuality in Large Language Model https://arxiv.org/pdf/2411.02433

needle-rs★ 109

600 KB WASM runtime for Cactus Compute's Needle AI tool-calling models, Needle 3, 2 and 1 from one build. Browser, Cloudflare Workers, Node.js, Python, C FFI and no_std. Token-exact with the JAX reference. No backend, no API key.

agent-infer★ 107

vLLM based inference framework for agentic workload.

sample-genai-on-eks-starter-kit★ 102

A comprehensive toolkit for deploying production-ready Generative AI infrastructure on Amazon EKS. Includes pre-configured components for: 🚀 AI Gateway (LiteLLM) 🤖 LLM Serving (vLLM, SGLang, Ollama) 📊 Vector Databases, 🔍 Embedding Models (TEI) 📈 Observability (Langfuse, Phoenix) etc. Fast-track your GenAI deployment with Kubernetes

MegatronApp★ 100

Toolchain built around the Megatron-LM for Distributed Training

qwen3-runtime★ 100

Rollout inference runtime for Agentic RL and multi-turn code agents

vibesys★ 99

Can AI Agents Build Bespoke Systems?

OpenAI-AWS-Lambda-Layer★ 97

OpenAI AWS Lambda Layer. Includes a pre-built package zip as well as a way to build your own.

llmariner★ 97

Extensible generative AI platform on Kubernetes with OpenAI-compatible APIs.

vLLM-Qwen3-3.5-3.6-chat-template-fix★ 95

This repo contains the critical chat_template.jinja fix for Qwen3/3.5/3.6 to work smoothly in agentic task.

langport★ 94

Langport is a language model inference service

sparkinfer★ 93

Fastest MoE/LLM inference runtime for consumer and edge Blackwell GPUs. SN74 on Gittensor.

agentic-api★ 93

Stateful API logic for agentic applications using vLLM

vllm-factory★ 88

Production inference for encoder models - ColBERT, GLiNER, ColPali, embeddings etc. - as vLLM plugins for online and in-process deployment

nlpcloud-python★ 86

NLP Cloud serves high performance pre-trained or custom models for NER, sentiment-analysis, classification, summarization, paraphrasing, intent classification, product description and ad generation, chatbot, grammar and spelling correction, keywords and keyphrases extraction, text generation, image generation, code generation, and more...

vibe-serve★ 81

Can AI Agents Build Bespoke LLM Serving Systems?

strands-sglang★ 81

SGLang model provider for Strands Agents.

strands-sglang★ 79

SGLang model provider for Strands Agents.

easy-model-deployer★ 77

Deploy open-source LLMs on AWS in minutes — with OpenAI-compatible APIs and a powerful CLI/SDK toolkit.

vllm-factory★ 77

Production inference for encoder models - ColBERT, GLiNER, ColPali, embeddings etc. - as vLLM plugins for online and in-process deployment

jevfire★ 77

JEV-inspired parallel decisions for CUDA LLMs. One context, many decisions. vLLM API, game-agent examples, and reproducible benchmarks.

small-gpu★ 76

Building a Tiny GPU to understand AI hardware engineering

qwen36-27b-single-5090★ 74

Validated recipe for serving Qwen3.6-27B on a single RTX 5090 — full OpenAI API, vision, tool calling, MTP spec-decode

SharpMind★ 69

SharpMind. A pure C# / .NET LLM engine — inference, training, and agent tooling in one solution.

SpecEyes★ 68

[ECCV 2026🔥] This is the official implementation of our paper "SpecEyes: Accelerating Agentic Multimodal LLMs via Speculative Perception and Planning"

vortex_torch★ 68

Vortex: Programmable Sparse Attention for Agents as Algorithm Designers

ignis★ 68

An inference engine for NVIDIA RTX card card family, built for a coding agent and its subagents, and a companion that listens, reads and draws: Qwen3.8 27B and 125B Flash-Next MoE, Qwen-Image-2.1, Qwen3-ASR on the CPU. OpenAI-compatible API and Playground in one binary.

flame★ 58

A distributed engine for elastic workload, e.g. AI Agent, RL, Quant

Qwen3-Coder-Next-NVFP4-GB10★ 54

transformers safetensors qwen3_next text-generation qwen3 nvfp4

modelship★ 54

Self-hosted, OpenAI-compatible inference for the agentic era: reasoning LLMs, universal tool calling, and the Responses API alongside embeddings, speech, and image models — many models sharing your GPUs, one gateway. Powered by Ray Serve.

vllm-awq4-qwen★ 53

vLLM Qwen 3.6-27B (AWQ-INT4) + DFlash speculative decoding on AMD Strix Halo (gfx1151 iGPU, 128 GB UMA, ROCm 7.13). 24.8 t/s single-stream, vision, tool calling, 256K context, OpenAI-compatible, Docker. Matches DGX Spark FP8+DFlash+MTP at a third of the cost. No CUDA.

gpu-ai-skills★ 50

Agent Skills for running, benchmarking, and profiling Hugging Face models on Intel GPUs — PyTorch, vLLM-XPU, and SGLang-XPU

MegaCapybara★ 50

The fastest inference engine for Qwen3.8-27B on the NVIDIA RTX 5090: up to 500 tokens/s for one agent and up to 2,000 tokens/s for many, contexts up to 1M tokens, and a launcher that shows what every setting costs. Windows and Linux.

nlpcloud-js★ 48

NLP Cloud serves high performance pre-trained or custom models for NER, sentiment-analysis, classification, summarization, paraphrasing, intent classification, product description and ad generation, chatbot, grammar and spelling correction, keywords and keyphrases extraction, text generation, image generation, code generation, and much more...

wsa★ 47

WSA(Websocket Streaming Agent) is a stream server target for mp4/h264 streaming over websocket

Spark-Studio★ 47

DGX Spark inference dashboard — vLLM, SGLang, llama.cpp, WebGPU & sparkrun with agent-powered Auto-Fix and speed optimization

clarifai-python★ 46

Experience the power of Clarifai’s AI platform with the python SDK. 🌟 Star to support our work!

snapllm★ 44

🔥 Alternative to Ollama — multi-model serving with sub-ms model switching · CPU-only 20B inference for Edge AI · llama.cpp + stablediffusion.cpp

claude★ 41

Deploy Anthropic Claude on Microsoft Foundry with `azd up` (Bicep or Terraform) and call it with the official Claude SDKs over Entra ID.

llm-scale-deploy-guide★ 40

An end-to-end pipeline to optimize and host LLM for 100K parallel queries

Qwen3-Coder-Next-int4-AutoRound★ 40

safetensors qwen3_next text-generation conversational arxiv:2309.05516 base_model:Qwen/Qwen3-Coder-Next

snapllm★ 39

🔥 Alternative to Ollama — multi-model serving with sub-ms model switching · CPU-only 20B inference for Edge AI · llama.cpp + stablediffusion.cpp

Qwen3-Coder-30B-A3B-Instruct-FP4★ 39

Model Optimizer safetensors qwen3_moe nvidia ModelOpt Qwen3

Qwen3-Coder-Next-NVFP4★ 37

safetensors qwen3_next NVFP4 quantized llm-compressor compressed-tensors

modelship★ 36

Self-hosted, OpenAI-compatible inference for the agentic era: reasoning LLMs, universal tool calling, and the Responses API alongside embeddings, speech, and image models — many models sharing your GPUs, one gateway. Powered by Ray Serve.

Qwen3-Coder-30B-A3B-DFlash★ 35

transformers safetensors qwen3 feature-extraction dflash speculative-decoding

LRAgent★ 34

Official implementation of LRAgent: Efficient KV Cache Sharing for Multi-LoRA LLM Agents

llm-server★ 33

🤖 Open-source LLM server (OpenAI, Ollama, Groq, Anthropic) with support for HTTP, Streaming, Agents, RAG (Deprecated check out Orchestra) ->

cuda-evolve-oss★ 32

Autonomous GPU kernel optimization system driven by AI agents.

IntelliServer★ 29

AI models as scalable microservices, enabling evaluation of LLMs and offering end-to-end functions such as chatbot, semantic search, image generation and beyond.

NVIDIA-Nemotron-3-Super★ 29

Controllable reasoning demos for NVIDIA Nemotron 3 Super (120B/12B MoE) — chat UI, CLI, API server, tool calling, budget sweep, and adaptive routing

Aquiles-Image★ 29

A high-performance, memory-efficient inference server for diffusion models (image/video generation), compatible with the OpenAI client

sirvir★ 27

Sirvir — model fleet manager & intelligence engine for Hermes Agent. Autonomous model lifecycle manager: local serving, benchmarking, auto-scaling, API fallback, HuggingFace scanning, creator tracking, backend optimization, budget monitoring.

More in Agent frameworks & infrastructure