Agent frameworks & infrastructure
Model serving & inference
Model serving and inference projects run and serve trained models rather than build them: vLLM provides a high-throughput, memory-efficient engine for LLM inference, while gpustack manages GPU clusters and exposes vLLM or SGLang backends as on-demand instances. They differ in the deployment concern each handles, from quantization and batching to OpenAI-compatible endpoints and cluster scheduling.
775 agents · most starred first
Top 100 of 775
A high-throughput and memory-efficient inference and serving engine for LLMs
Making large AI models cheaper, faster and more accessible
An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.
The Cloud Native AI Platform
Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.
A GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.
Distributed AI/LLM for the people. Share compute privately or publicly to power your agents and chat.
Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. Up to 4× faster than Apple's MLX (mlx-lm) on the same weights.
CSGHub is a brand-new open-source platform for managing LLMs, developed by the OpenCSG team. It offers both open-source and on-premise/SaaS solutions, with features comparable to Hugging Face. Gain full control over the lifecycle of LLMs, datasets, and agents, with Python SDK compatibility with Hugging Face. Join us! ⭐️
FEDML - The unified and scalable ML library for large-scale distributed training, model serving, and federated learning. FEDML Launch, a cross-cloud scheduler, further enables running any AI jobs on any GPU cloud or on-premise cluster. Built on this library, TensorOpera AI (https://TensorOpera.ai) is your generative AI platform at scale.
High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle
Open-source inference server and production cluster for all the models your agent needs.
LLM Inference Engine for Metal, CUDA and Vulkan.
⚡ Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for LLMs; run LLMs efficiently on Intel Platforms⚡
Vendor-agnostic orchestration for training, inference and agentic workloads across NVIDIA, AMD, TPU, and Tenstorrent on clouds, Kubernetes, and bare metal.
High-performance OpenAI and Anthropic compatible LLM inference server for Apple Silicon. Native MLX, continuous batching, multimodal models, MCP tool calling, and Claude Code support.
⚡ Langchain apps in production using Jina & FastAPI
Parallax is a distributed model serving framework that lets you build your own AI cluster anywhere
Jlama is a modern LLM inference engine for Java
⛓️ Serving LangChain LLM apps and agents automagically with FastApi. LLMops
Deploy any AI model, agent, database, RAG, and pipeline locally or remotely in minutes
[ICLR'26] The official code implementation for "Cache-to-Cache: Direct Semantic Communication Between Large Language Models"
A native .NET LLM inference engine and agent runtime for GGUF models. TensorSharp provides a console application, a web-based chatbot interface, iPhone App, and Ollama/OpenAI-compatible HTTP APIs for programmatic access. It supports Windows/MacOS/iOS/Linux with full GPU capability
A subset of PyTorch's neural network modules, written in Python using OpenAI's Triton.
Inference engine for Intel devices. Serve LLMs, VLMs, Whisper, Kokoro-TTS, Embedding and Rerank models over OpenAI endpoints.
🔥🔥 The extremely fast inference engine for decision model
KVarN is a native vLLM KV-cache quantization backend for your agents: 3-5x more context, throughput above FP16, and FP16-level accuracy. Calibration-free, one flag.
A command-line interface tool for serving LLM using vLLM.
A simple, fast and robust program-aware agentic inference system.
Low latency JSON generation using LLMs ⚡️
Agentic Kernel Optimization for All — automated GPU kernel optimization for any kernel, any hardware, any language
Local-first engineering agent that deploys, benchmarks and tunes vLLM/SGLang inference services on your own GPUs, with auditable run evidence.
💬 Chatbot web app + HTTP and Websocket endpoints for LLM inference with the Petals client
Winner 🏆 (Agent-only) MLSys 2026 - FlashInfer AI Kernel Generation Contest for the DeepSeek Sparse Attention (DSA) track with an average speedup of 34.93x
The production framework for Predictive and Generative AI. Serve any model as an API in one line, with OpenAI/Anthropic/Ollama-compatible endpoints, a built-in chat UI, and native MCP.
From PyTorch model to end-to-end TensorRT inference experience in two commands—AI-native, cross-platform, and built for the best possible user experience.
An agentic system that auto-optimizes LLM workloads on AMD GPUs.
gpt_server是一个用于生产级部署LLMs、Embedding、Reranker、ASR、TTS、文生图、图片编辑和文生视频的开源框架。
dgx-spark-qwen38 is an open-source installer that turns an NVIDIA DGX Spark, or any GB10 machine (ASUS, Dell, HP, Lenovo, MSI, GIGABYTE, Acer), into a private AI server in one command: measured model lanes, OpenAI and Anthropic APIs, and a local cockpit.
OpenAI compatible API for TensorRT LLM triton backend
A OpenAI API compatible REST server for llama.
A GPT-J API to use with python3 to generate text, blogs, code, and more
An agent harness that compiles a model into one provably-correct, self-retargeting CUDA megakernel and self-tunes it past cuBLAS at batch-1 LLM decode, paper: https://arxiv.org/abs/2606.09682
Higher performance OpenAI LLM service than vLLM serve: A pure C++ high-performance OpenAI LLM service implemented with GPRS+TensorRT-LLM+Tokenizers.cpp, supporting chat and function call, AI agents, distributed multi-GPU inference, multimodal capabilities, and a Gradio chat interface.
🪶 Lightweight OpenAI drop-in replacement for Kubernetes
SNDR Core Engine (Genesis) — vLLM runtime patch-overlay for Qwen3.6 + Gemma4 on consumer NVIDIA (Ampere sm_86, 2× A5000/3090). Qwen3.6-35B-A3B FP8 ~240 tok/s, 27B-int4 hybrid GDN+Mamba, Gemma4 26B/31B AWQ, 256K ctx. 321 patches: TurboQuant k8v4 KV, MTP/DFlash spec-decode, FULL cudagraph, hybrid GDN. vLLM pin dev424 + Control Center GUI.
Large Language Model (LLM) Inference API and Chatbot
SLED: Self Logits Evolution Decoding for Improving Factuality in Large Language Model https://arxiv.org/pdf/2411.02433
600 KB WASM runtime for Cactus Compute's Needle AI tool-calling models, Needle 3, 2 and 1 from one build. Browser, Cloudflare Workers, Node.js, Python, C FFI and no_std. Token-exact with the JAX reference. No backend, no API key.
vLLM based inference framework for agentic workload.
A comprehensive toolkit for deploying production-ready Generative AI infrastructure on Amazon EKS. Includes pre-configured components for: 🚀 AI Gateway (LiteLLM) 🤖 LLM Serving (vLLM, SGLang, Ollama) 📊 Vector Databases, 🔍 Embedding Models (TEI) 📈 Observability (Langfuse, Phoenix) etc. Fast-track your GenAI deployment with Kubernetes
Toolchain built around the Megatron-LM for Distributed Training
Rollout inference runtime for Agentic RL and multi-turn code agents
Can AI Agents Build Bespoke Systems?
OpenAI AWS Lambda Layer. Includes a pre-built package zip as well as a way to build your own.
Extensible generative AI platform on Kubernetes with OpenAI-compatible APIs.
This repo contains the critical chat_template.jinja fix for Qwen3/3.5/3.6 to work smoothly in agentic task.
Langport is a language model inference service
Fastest MoE/LLM inference runtime for consumer and edge Blackwell GPUs. SN74 on Gittensor.
Stateful API logic for agentic applications using vLLM
Production inference for encoder models - ColBERT, GLiNER, ColPali, embeddings etc. - as vLLM plugins for online and in-process deployment
NLP Cloud serves high performance pre-trained or custom models for NER, sentiment-analysis, classification, summarization, paraphrasing, intent classification, product description and ad generation, chatbot, grammar and spelling correction, keywords and keyphrases extraction, text generation, image generation, code generation, and more...
Can AI Agents Build Bespoke LLM Serving Systems?
SGLang model provider for Strands Agents.
SGLang model provider for Strands Agents.
Deploy open-source LLMs on AWS in minutes — with OpenAI-compatible APIs and a powerful CLI/SDK toolkit.
Production inference for encoder models - ColBERT, GLiNER, ColPali, embeddings etc. - as vLLM plugins for online and in-process deployment
JEV-inspired parallel decisions for CUDA LLMs. One context, many decisions. vLLM API, game-agent examples, and reproducible benchmarks.
Building a Tiny GPU to understand AI hardware engineering
Validated recipe for serving Qwen3.6-27B on a single RTX 5090 — full OpenAI API, vision, tool calling, MTP spec-decode
SharpMind. A pure C# / .NET LLM engine — inference, training, and agent tooling in one solution.
[ECCV 2026🔥] This is the official implementation of our paper "SpecEyes: Accelerating Agentic Multimodal LLMs via Speculative Perception and Planning"
Vortex: Programmable Sparse Attention for Agents as Algorithm Designers
An inference engine for NVIDIA RTX card card family, built for a coding agent and its subagents, and a companion that listens, reads and draws: Qwen3.8 27B and 125B Flash-Next MoE, Qwen-Image-2.1, Qwen3-ASR on the CPU. OpenAI-compatible API and Playground in one binary.
A distributed engine for elastic workload, e.g. AI Agent, RL, Quant
transformers safetensors qwen3_next text-generation qwen3 nvfp4
Self-hosted, OpenAI-compatible inference for the agentic era: reasoning LLMs, universal tool calling, and the Responses API alongside embeddings, speech, and image models — many models sharing your GPUs, one gateway. Powered by Ray Serve.
vLLM Qwen 3.6-27B (AWQ-INT4) + DFlash speculative decoding on AMD Strix Halo (gfx1151 iGPU, 128 GB UMA, ROCm 7.13). 24.8 t/s single-stream, vision, tool calling, 256K context, OpenAI-compatible, Docker. Matches DGX Spark FP8+DFlash+MTP at a third of the cost. No CUDA.
Agent Skills for running, benchmarking, and profiling Hugging Face models on Intel GPUs — PyTorch, vLLM-XPU, and SGLang-XPU
The fastest inference engine for Qwen3.8-27B on the NVIDIA RTX 5090: up to 500 tokens/s for one agent and up to 2,000 tokens/s for many, contexts up to 1M tokens, and a launcher that shows what every setting costs. Windows and Linux.
NLP Cloud serves high performance pre-trained or custom models for NER, sentiment-analysis, classification, summarization, paraphrasing, intent classification, product description and ad generation, chatbot, grammar and spelling correction, keywords and keyphrases extraction, text generation, image generation, code generation, and much more...
WSA(Websocket Streaming Agent) is a stream server target for mp4/h264 streaming over websocket
DGX Spark inference dashboard — vLLM, SGLang, llama.cpp, WebGPU & sparkrun with agent-powered Auto-Fix and speed optimization
Experience the power of Clarifai’s AI platform with the python SDK. 🌟 Star to support our work!
🔥 Alternative to Ollama — multi-model serving with sub-ms model switching · CPU-only 20B inference for Edge AI · llama.cpp + stablediffusion.cpp
Deploy Anthropic Claude on Microsoft Foundry with `azd up` (Bicep or Terraform) and call it with the official Claude SDKs over Entra ID.
An end-to-end pipeline to optimize and host LLM for 100K parallel queries
safetensors qwen3_next text-generation conversational arxiv:2309.05516 base_model:Qwen/Qwen3-Coder-Next
🔥 Alternative to Ollama — multi-model serving with sub-ms model switching · CPU-only 20B inference for Edge AI · llama.cpp + stablediffusion.cpp
Model Optimizer safetensors qwen3_moe nvidia ModelOpt Qwen3
safetensors qwen3_next NVFP4 quantized llm-compressor compressed-tensors
Self-hosted, OpenAI-compatible inference for the agentic era: reasoning LLMs, universal tool calling, and the Responses API alongside embeddings, speech, and image models — many models sharing your GPUs, one gateway. Powered by Ray Serve.
transformers safetensors qwen3 feature-extraction dflash speculative-decoding
Official implementation of LRAgent: Efficient KV Cache Sharing for Multi-LoRA LLM Agents
🤖 Open-source LLM server (OpenAI, Ollama, Groq, Anthropic) with support for HTTP, Streaming, Agents, RAG (Deprecated check out Orchestra) ->
Autonomous GPU kernel optimization system driven by AI agents.
AI models as scalable microservices, enabling evaluation of LLMs and offering end-to-end functions such as chatbot, semantic search, image generation and beyond.
Controllable reasoning demos for NVIDIA Nemotron 3 Super (120B/12B MoE) — chat UI, CLI, API server, tool calling, budget sweep, and adaptive routing
A high-performance, memory-efficient inference server for diffusion models (image/video generation), compatible with the OpenAI client
Sirvir — model fleet manager & intelligence engine for Hermes Agent. Autonomous model lifecycle manager: local serving, benchmarking, auto-scaling, API fallback, HuggingFace scanning, creator tracking, backend optimization, budget monitoring.