Agent frameworks & infrastructure

Model serving & inference

775 agents · most starred first

Model serving & inference, page 5 of 8

llm-autobatch

Automatic micro-batching for HTTP LLM calls and local PyTorch inference, backed by a Rust core.

llm-batch

Batch CLI tool for running batch inference with local LLMs

llm-cal

LLM inference hardware calculator — architecture-aware, engine-version-aware, honest-labeled.

llm-compression

Arithmetic compression using Llama LLM models

llm-coop-node

LLM Coop Node — connect your GPU to a coordinator for shared inference

llm-databricks

LLM access to Databricks model serving

llm-flexquant

a python library that lets you quantize any layer your llm

llm-gpu-lab

One GPU. Full LLM workflow. Real benchmarks. No cloud required.

llm-host

OpenAI-compatible inference server: Llama 3.1 8B + Whisper + Kokoro TTS exposed via ngrok

llm-infer

Multi-backend LLM client library, a from-scratch PagedAttention inference engine, and an OpenAI-compatible serve wrapper.

llm-inference

Large Language Models Inference API and Applications

llm-inference-engine

LLM inferencing utilities for multiple projects.

llm-inferencer

A collection of commonly used code

llm-launchpad

Personal LLM deployments made simple.

llm-layer-collector

Load and dispatch individual transformer layers from HuggingFace model checkpoints

llm-layer-prefetch

Stream transformer blocks layer-by-layer from disk to GPU with a pipelined prefetch queue (disk → CPU RAM → pinned RAM → GPU).

llm-mem

lllm-mem is a lightweight Python CLI tool designed to calculate the GPU VRAM requirements for models on Hugging Face

llm-mem-planner

Estimate LLM VRAM usage (weights/KV/activations) and recommend tp/dp/sp configs.

llm-partial-run

Microlib for partial deployment of LLMs

llm-platform

LLM serving platform

llm-quantkit

cli tool for downloading and quantizing LLMs

llm-resource-planner

CLI tool to estimate VRAM requirements and GPU fit for Hugging Face LLMs without downloading model weights

llm-serve

An LLM inference solution to quickly deploy productive LLM service

llm-server
llm-serving

A package to perform LLM inference in Solenium/Unergy HQ

llm-skeleton

Probe → Plan → Load. Universal model loader for multi-GPU LLM deployment.

llm-swarm
llm-tokenizer

Tokenizer utilities for LLM inference

llm-tokenizers

LLM tokenizers tools

llm-vm

An Open-Source AGI Server for Open-Source LLMs

llm-weights-mmap

Microlib for memory-mapping LLM (large language models) weights

llm-x-py

Python library for precise, hardware-aware inference memory estimation of language models.

llm-xs

A module for creation of endpoint for the open-sourced LLMs

llmaz

☸️ Easy, advanced inference platform for large language models on Kubernetes

llmbo-bedrock

Large Language Model Batch Operations

llmboost-hub

A lightweight CLI tool for managing LLMBoost™ model images and environments.

llmcompressor

A library for compressing large language models utilizing the latest techniques and research in the field for both training aware and post training techniques. The library is designed to be flexible and easy to use on top of PyTorch and HuggingFace Transformers, allowing for quick experimentation.

llmcompressor-nightly

A library for compressing large language models utilizing the latest techniques and research in the field for both training aware and post training techniques. The library is designed to be flexible and easy to use on top of PyTorch and HuggingFace Transformers, allowing for quick experimentation.

llmdeploy

triton inference service of llama

llmexport

llmexport: A toolkit to export llm to onnx or mnn.

llmflux

CLI tool for running LLM batch processing jobs on HPC systems

llmgrad

A C++ AI engine with Python bindings

lmcache-vllm

lmcache_vllm: LMCache's wrapper for vllm

londonaicentre-llama-serve

Serve llama models locally

matorage

matorage is Matrix or Tensor(multidimensional matrix) Object Storage with high availability distributed systems for Deep Learning framework.

medusa-llm

Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads

min-llm-server-client

A minimal API server for local HuggingFace LLMs or VLLM LLMs

mindspeed-llm

MindSpeed for LLMs of Ascend

mini-vllm

An educational implementation of an inference engine

minillm

Simple inference for large language models

minima-llm

Minimal async LLM backend with caching and batch execution

minty-infra-storage

Storage infrastructure for the Minty framework

mirage-project

Mirage: A Multi-Level Superoptimizer for Tensor Algebra

mistralrs

Fast, flexible LLM inference.

mistralrs-accelerate

Fast, flexible LLM inference.

mistralrs-cuda

Fast and easy LLM serving.

mistralrs-metal

Fast, flexible LLM inference.

mistralrs-mkl

Fast, flexible LLM inference.

mlc-llm-cpu

Machine learning for systems

mlc-llm-cu117

Machine learning for systems

mlc-llm-cu121

Machine learning for systems

mlc-llm-cu122

Machine learning for systems

mlc-llm-cu124

Machine learning for systems

mlc-llm-cu128

Machine learning for systems

mlc-llm-cu130

Machine learning for systems

mlc-llm-nightly-cpu

Machine learning for systems

mlc-llm-nightly-cu117

Machine learning for systems

mlc-llm-nightly-cu121

Machine learning for systems

mlc-llm-nightly-cu122

Machine learning for systems

mlc-llm-nightly-cu124

Machine learning for systems

mlc-llm-nightly-cu128

Machine learning for systems

mlc-llm-nightly-cu130

Machine learning for systems

mlc-llm-nightly-rocm56

Machine learning for systems

mlc-llm-nightly-rocm61

Machine learning for systems

mlc-llm-nightly-rocm62

Machine learning for systems

mlc-llm-rocm56

Machine learning for systems

mlc-llm-rocm61

Machine learning for systems

mlc-llm-rocm62

Machine learning for systems

mlx-llm-server

server to serve mlx model as an OpenAI compatible API

mlx-openai-server

A high-performance API server that provides OpenAI-compatible endpoints for MLX models.

mlxllm-engine

MLX-optimized vLLM v1 implementation for Apple Silicon

modelz-llm

LLM unified service

mycellm

Distributed LLM inference across heterogeneous hardware

mypy-boto3-kinesis-video-webrtc-storage

Type annotations for boto3 KinesisVideoWebRTCStorage 1.44.0 service generated with mypy-boto3-builder 8.12.0

nano-vllm-fork

a lightweight vLLM implementation built from scratch

nano-vllm-npu

a lightweight vLLM implementation built from scratch and runs on NPU.

nano-vllm-voxcpm

VoxCPM inference engine based on Nano-vLLM

nano-vllm-voxcpm-hifi

VoxCPM inference engine based on Nano-vLLM

nanoflow-llm

NanoFlow v0.7: The Universal Sparse Streaming Inference Engine

nanoflow-llm-v7

NanoFlow v0.7 Sparse Engine

next-gen-ui-llama-stack-embedded

Embedded Llama-Stack server inference for Next Gen UI Agent

nextai-vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

nm-vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

npu-vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

ollama-fastapi

FastAPI wrapper for running Ollama locally

ollama-ipex

Local control plane for the IPEX-backed Ollama runtime.

ollm

LLM Inference for Large-Context Offline Workloads

ommi-llm

Run 70B+ LLMs on consumer GPUs with layer-wise inference

oomllama

Efficient LLM inference engine with .oom format - from-scratch Rust, SafeTensors/GGUF converters

open-gpt-torch

An open-source cloud-native of large multi-modal models (LMMs) serving framework.

More in Agent frameworks & infrastructure