Agent frameworks & infrastructure

Model serving & inference

775 agents · most starred first

Model serving & inference, page 2 of 8

llm-inference-calculator★ 27

A static performance model for LLM inference: given (model, GPU, interconnect, workload), instantly answer VRAM feasibility, latency(TTFT, TPOT, ITL), and throughput.

agentic-kv-cache★ 26

Reproducing agentic KV-cache policy claims on real traces. 68k requests from 393 Claude Code sessions. LRU is harder to beat than the papers suggest.

clarifai-nodejs★ 25

Experience the power of Clarifai’s AI platform with the nodejs SDK. 🌟 Star to support our work!

aigc_serving★ 24

aigc_serving lightweight and efficient Language service model reasoning

seq2seq_ncnn★ 24

the C++ version of Seq2Seq with ncnn

frokenizer★ 24

A zero-allocation, header-only C++ BPE tokenizer for Qwen, built for maximum inference throughput.

contextcache★ 22

Tool schemas get re-prefilled on every LLM request. ContextCache compiles them into a KV cache once, stores it on disk and reuses it across users and sessions — TTFT stays near 200ms with 50 tools instead of 5.6s (Qwen3-8B, RTX 3090 Ti).

hpc-agentic-recipes★ 22

Local Agentic Model Deployment on AI Cluster

chorus-llm★ 22

A local LLM runtime for many-agent applications, built on llama.cpp. One model, shared inference, many agents, embedded. C++/C, Vulkan, Metal, Godot.

guidance-for-scalable-model-inference-and-agentic-ai-on-amazon-eks★ 20

Comprehensive, scalable ML inference architecture using Amazon EKS, leveraging Graviton processors for cost-effective CPU-based inference and GPU instances for accelerated inference. Guidance provides a complete end-to-end platform for deploying LLMs with agentic AI capabilities, including RAG and MCP

atra★ 20

An open source NLP as a service project focused on providing state of the art systems with ease. Training and inference by simple docker commands

tinycoral★ 19

tinygrad on Google's Coral Edge TPU: reverse-engineered ISA, our own code generator and USB driver. A whole transformer runs on the stick. Written by AI agents.

Aris-AI-Model-Server★ 18

An OpenAI Compatible API which integrates LLM, Embedding and Reranker. 一个集成 LLM、Embedding 和 Reranker 的 OpenAI 兼容 API

profine-cli★ 18

Profine automatically profiles and optimizes PyTorch training jobs on real GPUs, delivering measurable speedups and lower GPU costs before teams waste days tuning configs by hand.

rMLX★ 18

Native Rust, single-binary MLX inference server for Apple Silicon. OpenAI/Anthropic-compatible LLM serving — text, vision, audio, embeddings — with the widest weight×KV quantization matrix of any MLX server. No Python, no GGUF.

programmable-kv★ 17

Models Take Notes at Prefill: KV Cache Can Be Editable and Composable (arXiv:2606.17107) — paper, code, results, and interactive companion.

llm-engineering-platform★ 17

A production-oriented LLM engineering platform with OpenAI-compatible serving, streaming, observability, evaluation, reproducible experiments, and deterministic RAG.

LLM_Quantization★ 16

streamlit region:us

LLM_Quantization★ 16

streamlit region:us

MLOps★ 15

Miscellaneous codes and writings for MLOps

grid★ 15

AI Grid Network

haif★ 14

Production-ready microservices framework for AI inference over RPC. It provides a Gateway for client requests, an Orchestrator that schedules work, a Registry for model metadata, Workers that run inference, and a full observability stack (Prometheus, Grafana, Loki, Jaeger) — all wired together with Docker Compose.

dgx_spark★ 14

Multi-model LLM serving for NVIDIA DGX Spark with vLLM, web UI, and tool calling

aleph★ 14

A science inference cluster — every model, from protein folding to LLMs, behind one OpenAI/Anthropic-compatible endpoint and a single key. Built to run beside HPC so agentic AI researchers can drive simulations and model inference together. Self-deploying on RKE2 with KServe/Knative + HAMi GPU.

Cuda-Compute-OSS★ 13

CCO is an open-source system for managing, validating, and improving GPU.

SigmaStudio★ 12

Sigma Studio is a modular AI platform for orchestrating local and distributed AI workloads. Manage LLMs, multimodal models, image generation, training, inference and AI agents across GPUs, CPUs and edge devices, with intelligent resource-aware model selection.

ask-poddy★ 11

Ask Poddy: Run Open Source LLMs and Embeddings as OpenAI-Compatible Serverless Endpoints (Tutorial)

a11★ 11

A streaming action runtime for AI agents, model serving, and multimodal APIs

TChatBot-API★ 10

A Flask REST API to serve trained ChatBots using Tensorflow Serving and Docker Containers

kaggle-llm-api★ 10

🤖 Comprehensive solution for running Ollama/vLLM API servers in Kaggle notebooks with OpenAI-compatible endpoints

enterprise-ai-solutions★ 10

Production-ready platform for deploying OpenAI-compatible LLM inference on Kubernetes. A one-command installer for enterprise generative AI on Intel® Xeon®, with vLLM and other serving engines, secure Keycloak authentication, and built-in observability - from bare metal to a scalable AI endpoint.

fastapi-stream-lease★ 10

Distributed stream and SSE concurrency lease manager for FastAPI and Starlette backed by atomic Redis Lua scripts.

langserve-template★ 9

LangServe 템플릿 코드

nlpcloud-ruby★ 9

NLP Cloud serves high performance pre-trained or custom models for NER, sentiment-analysis, classification, summarization, paraphrasing, intent classification, product description and ad generation, chatbot, grammar and spelling correction, keywords and keyphrases extraction, text generation, image generation, code generation, and much more...

EVOKE★ 9

EVict and recOver KV cache Entries. Selective KV cache eviction and recovery for long-context LLM inference.

Lumen★ 9

LLM inference in Rust - Metal & CUDA

agent-gpu-calculator★ 9

Single-file GPU/RAM sizing calculator for LLM agent sessions on vLLM: KV-cache offload to RAM, prefix caching, MLA/DSA and hybrid models, TP chosen per model × GPU pair (H100–B300). Runs in the browser, no dependencies.

Jolt★ 9

Judgment On Low-latency Thresholds. A under-100M-parameter reflex layer that decides agent steps in <10ms and escalates to an LLM only when uncertain.

transAPI★ 8

OpenAI compatible API purely based on Transformers

omlx★ 8

🤖 Optimize LLM inference on Mac with continuous batching and SSD caching managed from your menu bar for efficient performance.

Ovis2-34B-GPTQ-Int4★ 8

gradio region:us

genpark-speculative-decoding-draft-verifier-skill★ 8

GenPark AI Agent Skill - Speculative decoding simulation engine coordinating small draft model speculative token generation and large target model parallel rejection sampling verification.

genpark-prefix-caching-kv-state-reuse-skill★ 8

GenPark AI Agent Skill - Radix tree prefix caching simulator for prompt templates and agent system instructions, maximizing KV-cache hit rates and cutting TTFT latency.

kime★ 8

Typed decisions over text in milliseconds. A Rust inference engine and server that answers choice, score and yes or no questions with calibrated probabilities, compatible with Jev and Laya. The bar is 10x faster on every benchmark at equal or better accuracy.

genpark-speculative-decoding-verifier-skill★ 8

Draft-target model speculative decoding verification engine with rejection sampling, acceptance rate telemetry, and dynamic speedup ratio estimation.

genpark-tensor-core-wmma-emulator-skill★ 7

Tensor Core Warp Matrix Multiply-Accumulate (WMMA) micro-tile systolic array emulator supporting outer-product GEMM and shared memory bank conflict reduction.

genpark-sparse-spmm-csr-kernel-skill★ 7

Compressed Sparse Row (CSR) Sparse Matrix-Dense Matrix Multiplication (SpMM) acceleration kernel with row-pointer parallelism and load-balanced binning.

genpark-flash-attention-tiling-kernel-skill★ 7

Online Softmax Tiling FlashAttention kernel executing bounded SRAM query/key/value block updates with running max and sum rescaling for O(1) memory footprint.

genpark-gpu-warp-divergence-simulator-skill★ 7

SIMT GPU warp execution simulator tracking active thread divergence masks, reconvergence points, and warp vote intrinsics (__ballot_sync, __all_sync, __any_sync).

genpark-kv-cache-paged-attention-allocator-skill★ 7

Virtual paged attention KV-cache memory manager with logical block mapping, non-contiguous physical page allocation, and zero-copy fragmentation tracking.

ljqinfer★ 7

Deploy DeepSeek-V4.1 (dsv4.1) on Ascend 910B / 910B3 (NPU): a hardware-specialized LLM inference engine with W4A8, DSpark speculative decoding, 128K+ context and 8,700+ tok/s prefill. Also supports NVIDIA A100/A800. 昇腾910B 部署 DeepSeek-V4.1-Flash。

jimbomesh-holler-server★ 6

Open source AI inference server with Model Marketplace, Document RAG, and OpenAI-compatible API

oshepherd★ 6

Oshepherd: Ollama instances Orchestration

LLM_Quantization★ 6

streamlit region:us

dgx-spark-qwen38-flash-agent★ 6

Qwen3.8-Flash-Next as Cogni-Brain on NVIDIA DGX Spark (GB10): HashK GPU PLE + SGLang NEXTN, 36.8 tok/s code, 100/100 tool-eval, 262K context.

llm-inference-engineering★ 6

Production code for the book "LLM Inference Engineering: Latency, Cost and Reliability at Scale"

pulsarlm★ 6

An LLM inference engine for persistent chat bots and agents. It allows models to forget old and unimportant tokens to keep context small and inference fast.

openPhalanx★ 6

open source project for making GPU based agent calls easy.

ritual-node★ 5

Ritual AI ❖ ❖ A Comprehensive Guide Running Infernet Node

AIMA★ 5

AI-Inference-Managed-by-AI: Go binary for managing AI inference on edge devices

docker-functionary★ 5

Ready-to-deploy Docker image for Functionary LLM served as an OpenAI-Compatible API.

o1-server★ 5
tiktoken-server★ 5

Docker container to expose the OpenAI tokenizer as a REST service

LLM-deployment-zerocost-api★ 5

docker region:us

hf-llm-api★ 5

docker region:us

llm-cache-management-code★ 5

Python implementations, C++ structures, and CUDA GPU kernels that demonstrate the core algorithms, memory allocators, quantization schemes, disaggregation transports, and latency profilers used in modern LLM serving engines like vLLM, SGLang, TensorRT-LLM, llama.cpp, NVIDIA Dynamo, and LMCache.

openai-workers-ai★ 5

A Cloudflare Workers-based, OpenAI-compatible API project that provides Whisper-based audio transcription capabilities. Built with TypeScript and the Hono framework, this project offers a fast, scalable, and serverless solution for integrating AI-driven speech recognition into your applications.

enterprise-inference★ 5

Enterprise-grade, high-performance LLM serving on Kubernetes. OpenAI-compatible inference powered by vLLM and OpenVINO™ Model Server behind a unified AI gateway, with NUMA-aware CPU optimization for Intel® Xeon® - production model serving, routing, and security built in.

cloudcdn.pro★ 5

Self-hosted, multi-tenant CDN and digital asset management (DAM) platform built on Cloudflare Workers and Pages. Features real-time image optimization, signed URLs, Workers AI vision, semantic search, WebAuthn passkeys, OpenAPI 3.1, and an MCP server for autonomous AI agents.

vlmhub★ 5

A lightweight, config-driven framework for unified vision-language model inference across local and cloud backends, built on LiteLLM and Transformers.

silicon-llm★ 4

OpenAI/Anthropic compatible LLM inference server for Apple Silicon

geofractal★ 4

Wide-model collective ensemble system with fractal, geometric, and heavy compilation optimizations.

torched-django★ 4

Deploying pytorch AI model with Django

quant-llms-text-generation-comparison★ 4

gradio region:us

garuda★ 4

Rust MoE inference runtime with tiered expert storage — loads real GGUF models; speaks OpenAI, Ollama, Anthropic, llama.cpp & TGI APIs

omniedge★ 4

Autonomous post-quantum-secured decentralized edge-AI mesh with agentic AI, RAG, and CNN vision

hipfire-kvmem★ 4

hipfire × KVMem —— 把 KVMem 的分页/分层 KV 移植进 hipfire 这个 RDNA 原生 Rust 引擎,让 262144 上下文在 24 GB 显卡上跑起来;agent 负载实测中位 32 tok/s。Paged/tiered KV (KVMem) ported into the RDNA-native Rust engine hipfire.

FerrisRes★ 3

Rust-native AI inference & training engine. Block AttnRes linear-time transformer with profile-driven GPU dispatch, TurboQuant, GGUF/SafeTensors, LoRA, multimodal (vision/audio/video), FerrisRes Armor security. 39K LOC, 951 tests. Adapts to any GPU via wgpu — Intel iGPU to H100. No Python.

vertex-ai-dotnet-csharp★ 3

Vertex AI Setup in C#

anthropic-fastapi-service★ 3

This is a FastAPI service that provides a RESTful API for the Anthropic Claude AI model using anthropic-sdk-python.

batchi★ 3

🚀 Enable efficient deep learning serving with dynamic batching that optimizes batch size and reduces latency while isolating invalid requests.

swarmkv★ 3

C++ inference runtime for llama.cpp that shares a single document KV-cache prefill across multiple analytical branches via snapshot fan-out. Eliminates redundant GPU compute and dramatically reduces TTFT in DAG-based multi-agent pipelines.

LLM_Quantization★ 3

streamlit region:us

gemma-4-26B-A4B-it-qat-assistant-MTP-Q8_0.gguf★ 3

gguf llama.cpp speculative-decoding mtp gemma4-assistant draft-model

adk-litert★ 3

Low-latency in-memory, in-process LLM engine for scalable and real-time inference in Go. Repurposing LiteRT-LM & Gemma 3 for backend systems and Google ADK v2 agents.

llm-calculator★ 2

LLM Training / Inference Calculator

LLM-Model-VRAM-Calculator★ 2

static region:us

LLM-memory-calculator★ 2

gradio region:us

gptq-pro-smoke24-agentic-3090★ 2

static terminal-bench agentic-workload gptq-pro vllm rtx-3090 benchmark region:us

adaptive-llm-quantization★ 2

Agentic framework for hardware-aware, layer-wise LLM quantization using sequential optimization and on-device benchmarking.

devproofai★ 2

Managed agents on Kubernetes + Self-hosted LLM serving. Own your scalable AI.

vBuf★ 2

A high-performance, zero-copy binary serialization format optimized for SIMD instructions and mmap-readability.

mojo-kernels★ 2

Clean-room Mojo kernels as drop-in accelerators for popular Python/TS libraries — measured 70x-8,700x with bit-exact parity and pure-language fallbacks. bm25, Fuse.js fuzzy search, cclib electron-density grids. Siblings: thyn-ai/algenta-sdk, thyn-ai/algenta-integrations.

Kimi-K2.7-Code-UD-Q4_K_XL-layers★ 1

mesh-llm gguf layer-package skippy distributed-inference local-inference

Qwen3-Coder-Next.w4a16★ 1

safetensors qwen3_next INT4 quantized llm-compressor compressed-tensors

gemma-4-31B-it-assistant-MTP-GGUF★ 1

gguf mtp draft-model assistant speculative-decoding gemma4

gemma-4-12B-it-qat-assistant-q4_0-exact-gguf★ 1

gguf gemma4 llama.cpp q4_0 qat speculative-decoding

groq

Groq review: compare features, pricing, use cases, access model, and alternatives for this Model Serving agent in 2026. AI accelerator company revolutionizing inference with ultra-fast Language Processing Units (LPUs)

Together AI

Power your AI apps with fast open-source models and GPUs.

Groq

Get ultra-fast, low-cost AI inference with custom processing chips.

More in Agent frameworks & infrastructure