Agent frameworks & infrastructure
Model serving & inference
775 agents · most starred first
Model serving & inference, page 8 of 8
A Python application toolkit for GPU-native data tools.
vLLM native KV-transfer adapter backed by kvblockd (KVConnectorBase_V1 + SecondaryTierManager)
High-performance hybrid LLM inference engine with Adaptive Non-Uniform 2-Bit Quantization (Lloyd-Max + Fused Triton INT2 GEMM), RLM Context-as-a-Variable engine, 98.2% logit parity, and Zero-RAM quantizer.
Typed, async Python SDK for AWS Bedrock Runtime.
vLLM architecture plugin for Exionos X1
OpenAI-compatible HTTP API for the Needle 2 tool-calling model
Local-first LLM serving for software that has to work: a resident engine, typed errors, validated extraction and an OpenAI-compatible API — on hardware you don't control, with data that can't leave.
Package for deploy LLM Model Service
Distributed LLM inference — pool GPUs across multiple devices to run models no single machine can handle
An integration between the Hetzner Inference API and Haystack
Beam search plugin for vLLM v1 with CoW block table forking
Confidently navigate LLM deployments from concept to production
TLLM runs 70B large language models on a single 4GB GPU without quantization, distillation or pruning. 405B Llama 3.1 on 8GB, DeepSeek-V3 671B on ~12GB, Kimi K3 2.8T on under 4GB, Qwen3.8-27B on 3.3GB.
Estimate LLM training and serving memory from model configs; measured training evidence covers one Tesla T4.
Run Fortel machine-learning jobs on your own machine.
Run bigger LLMs on smaller GPUs through intelligent, asynchronous layer streaming.
CLM: context-lifecycle management perception plugin for vLLM-HUST (request-lifecycle event sink)
DLA: decode-length-aware preemption & admission for vLLM-HUST
Benchmark-driven configuration optimization for vLLM inference servers
SBW watermarking logits processor for vLLM
Spark2_5 support for vLLM, maintained out of tree
Extension vllm created by CME
llama.cpp plugin for flyte
Lightweight, modular, hardware-aware LLM inference engine & empirical research testbed.
Low-downtime hot migration for vLLM inference services, via monkey-patching and a plugin (no vLLM source changes).
Out-of-tree speculative decoding plugin for vLLM-HUST on Ascend
A vLLM-style, production-grade LLM inference server built natively for Windows (single GPU, single machine)
Geodd provides fast, steady AI model hosting through serverless inference and dedicated GPUs. It supports OpenAI-compatible tools, regional deployment, usage tr
Sub-layer fragment placement for GPUs too small to hold one full transformer layer -- capacity-proportional splitting, column/row-parallel execution, verified against an unsplit reference.
Package name reserved — implementation coming soon
vLLM plugin for DuplexJev speech-decision models (Qwen3-ASR encoder + Ultravox-style projector).
One-shot migration of Kimi K3 inference from Fireworks to Amazon Bedrock for Python repositories
Any chat model, any engine (SGLang, vLLM, transformers, MLX) as a Jev-compatible probability decision service
OpenAI-compatible HTTP API for the deepgrove/maple-preview reasoning model
Type-Safe Decoding for Autoregressive LLMs
TriAttention KV-cache compression plugin for vLLM Ascend
Fail-closed runtime support for versioned Ascend quantized model artifacts
Runtime-loadable INT8 KV-cache attention backend for vLLM-Ascend-HUST
vLLM-HUST plugin carriers for vllm-ascend: decode FIA de-mask, RoPE variant fixes, zero-cost wiring, MLP chunking and inert split-batch planning
Modular graph and replay optimizations for vLLM Ascend
A high-throughput and memory-efficient inference and serving engine for LLMs
BidKV — CompressionBid protocol layer for KV cache scheduling primitives
Migration package for OPset, the vLLM-HUST operator optimization bundle
Pluggable kernel replacements and operator fusions for the vLLM-HUST Ascend backend
Zero-copy CUDA/MACA IPC weight pre-load cache for vLLM instant restart
AWS Bedrock streaming model provider for @onivoro/server-agentic, with pluggable per-model wire protocols
Deterministic high-precision VRAM, GQA KV-Cache, MoE Memory Bandwidth, and llama.cpp layer offloading profiler for 2026 AI models and silicon.
The kern runtime for Python, and the plugins that run a kern manifest as a vLLM or SGLang model
An integration package connecting FlexAI and LangChain
Tensor math, machine learning, deep learning and model inference helpers: reusable functions and best practices.
LLM VRAM calculator and cheapest AWS GPU recommender for vLLM, DeepSeek-R1, Llama 70B, and Qwen. JSON output for AI agents.
Fast MiMo-V2.5-ASR inference and OpenAI-compatible serving with vLLM
Pion prompt cache for mlx-lm on Apple Silicon — reuse a prefix's K/V across processes and restarts
Bounded native Rust key value cache and attention primitives for Python LLM pipelines
vLLM plugin that serves sabi-noema-mla (shared pre-RoPE rotary MLA, rank 512, on the 16 full-attention layers of a Qwen3.5/3.8 hybrid)
Expose a local LLM runtime as a reliable HTTP/WebSocket service.
TensorRT Edge-LLM model build and inference package
sllm: a high-performance LLM inference engine (placeholder; first release coming soon)
SLA-aware request scheduling for LLM serving, with a mock engine, an API server and benchmark tools
IQuest-Q1 models and parsers for vLLM
vLLM plugin: keeps the Flash-Next PLE table out of the vLLM process (per-machine agent with a shared cache, local file and/or central server)
A high-throughput and memory-efficient inference and serving engine for LLMs
XTU (X Transformers Unity) — a mainline-vLLM hybrid MoE plugin (CPU experts + GPU attention/long prefill). Bundles the xiaotu CPU MoE engine (AVX512-VNNI/BF16: MXFP4/FP8/INT4/BF16) so no-AMX x86 hosts (AMD EPYC) get usable CPU MoE backends through FusedMoEFactory.
Volcengine storageebs SDK for Python
Pull, run and seal LLMs and diffusion models on SGLang and vLLM.
CLI tool to audit real cloud GPU TCO, spot interruption penalty, and generative multimodal AI costs.
vLLM plugin: serve EXL3 (exllamav3 trellis) checkpoints and the model-family fixes they need, with no patch to vLLM
CLI and programmatic API to launch LLM inference endpoints on Slurm, using simple configuration files
vLLM Model Runner V2 logits processor for the text watermark plus an ack-header middleware.
Telys external KV/prefix-cache connector for vLLM (KVConnectorBase_V1)
vLLM plugin for Lernex's Metis-1.6 models: the Metis16ForCausalLM architecture plus the metis reasoning and tool-call parsers
Run model services on Google Colab with managed tunnels and authentication