Agent frameworks & infrastructure
Model serving & inference
775 agents · most starred first
Model serving & inference, page 5 of 8
Automatic micro-batching for HTTP LLM calls and local PyTorch inference, backed by a Rust core.
Batch CLI tool for running batch inference with local LLMs
LLM inference hardware calculator — architecture-aware, engine-version-aware, honest-labeled.
Arithmetic compression using Llama LLM models
LLM Coop Node — connect your GPU to a coordinator for shared inference
LLM access to Databricks model serving
a python library that lets you quantize any layer your llm
One GPU. Full LLM workflow. Real benchmarks. No cloud required.
OpenAI-compatible inference server: Llama 3.1 8B + Whisper + Kokoro TTS exposed via ngrok
Multi-backend LLM client library, a from-scratch PagedAttention inference engine, and an OpenAI-compatible serve wrapper.
Large Language Models Inference API and Applications
LLM inferencing utilities for multiple projects.
A collection of commonly used code
Personal LLM deployments made simple.
Load and dispatch individual transformer layers from HuggingFace model checkpoints
Stream transformer blocks layer-by-layer from disk to GPU with a pipelined prefetch queue (disk → CPU RAM → pinned RAM → GPU).
lllm-mem is a lightweight Python CLI tool designed to calculate the GPU VRAM requirements for models on Hugging Face
Estimate LLM VRAM usage (weights/KV/activations) and recommend tp/dp/sp configs.
Microlib for partial deployment of LLMs
LLM serving platform
cli tool for downloading and quantizing LLMs
CLI tool to estimate VRAM requirements and GPU fit for Hugging Face LLMs without downloading model weights
An LLM inference solution to quickly deploy productive LLM service
A package to perform LLM inference in Solenium/Unergy HQ
Probe → Plan → Load. Universal model loader for multi-GPU LLM deployment.
Tokenizer utilities for LLM inference
LLM tokenizers tools
An Open-Source AGI Server for Open-Source LLMs
Microlib for memory-mapping LLM (large language models) weights
Python library for precise, hardware-aware inference memory estimation of language models.
A module for creation of endpoint for the open-sourced LLMs
☸️ Easy, advanced inference platform for large language models on Kubernetes
Large Language Model Batch Operations
A lightweight CLI tool for managing LLMBoost™ model images and environments.
A library for compressing large language models utilizing the latest techniques and research in the field for both training aware and post training techniques. The library is designed to be flexible and easy to use on top of PyTorch and HuggingFace Transformers, allowing for quick experimentation.
A library for compressing large language models utilizing the latest techniques and research in the field for both training aware and post training techniques. The library is designed to be flexible and easy to use on top of PyTorch and HuggingFace Transformers, allowing for quick experimentation.
triton inference service of llama
llmexport: A toolkit to export llm to onnx or mnn.
CLI tool for running LLM batch processing jobs on HPC systems
A C++ AI engine with Python bindings
lmcache_vllm: LMCache's wrapper for vllm
Serve llama models locally
matorage is Matrix or Tensor(multidimensional matrix) Object Storage with high availability distributed systems for Deep Learning framework.
Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads
A minimal API server for local HuggingFace LLMs or VLLM LLMs
MindSpeed for LLMs of Ascend
An educational implementation of an inference engine
Simple inference for large language models
Minimal async LLM backend with caching and batch execution
Storage infrastructure for the Minty framework
Mirage: A Multi-Level Superoptimizer for Tensor Algebra
Fast, flexible LLM inference.
Fast, flexible LLM inference.
Fast and easy LLM serving.
Fast, flexible LLM inference.
Fast, flexible LLM inference.
Machine learning for systems
Machine learning for systems
Machine learning for systems
Machine learning for systems
Machine learning for systems
Machine learning for systems
Machine learning for systems
Machine learning for systems
Machine learning for systems
Machine learning for systems
Machine learning for systems
Machine learning for systems
Machine learning for systems
Machine learning for systems
Machine learning for systems
Machine learning for systems
Machine learning for systems
Machine learning for systems
Machine learning for systems
Machine learning for systems
server to serve mlx model as an OpenAI compatible API
A high-performance API server that provides OpenAI-compatible endpoints for MLX models.
MLX-optimized vLLM v1 implementation for Apple Silicon
LLM unified service
Distributed LLM inference across heterogeneous hardware
Type annotations for boto3 KinesisVideoWebRTCStorage 1.44.0 service generated with mypy-boto3-builder 8.12.0
a lightweight vLLM implementation built from scratch
a lightweight vLLM implementation built from scratch and runs on NPU.
VoxCPM inference engine based on Nano-vLLM
VoxCPM inference engine based on Nano-vLLM
NanoFlow v0.7: The Universal Sparse Streaming Inference Engine
NanoFlow v0.7 Sparse Engine
Embedded Llama-Stack server inference for Next Gen UI Agent
A high-throughput and memory-efficient inference and serving engine for LLMs
A high-throughput and memory-efficient inference and serving engine for LLMs
A high-throughput and memory-efficient inference and serving engine for LLMs
FastAPI wrapper for running Ollama locally
Local control plane for the IPEX-backed Ollama runtime.
LLM Inference for Large-Context Offline Workloads
Run 70B+ LLMs on consumer GPUs with layer-wise inference
Efficient LLM inference engine with .oom format - from-scratch Rust, SafeTensors/GGUF converters
An open-source cloud-native of large multi-modal models (LMMs) serving framework.