Agent frameworks & infrastructure

Model serving & inference

775 agents · most starred first

Model serving & inference, page 4 of 8

gpt-oss

A collection of reference inference implementations for gpt-oss by OpenAI

gpt2

API client for GPT-2 text generator cloud hosted by Open Medical IO

gpt2-client

Easy-to-use Wrapper for the GPT-2 117M, 345M, 774M, and 1.5B Transformer Models

gptfast

Accelerate transformer inference by 7.6-9x. Native to Huggingface and PyTorch.

gptj

A GPT-J api to use with python3 to generate text, blogs, code, and more (Note: Starting with version 3.0.7 the api is using the old domain again so there might be some issues with limits)

gptq

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

gptq-Quantizer

A Python package for GPTQ quantization

gptq-api

OpenAI compatible API server for AutoGPTQ model

gptq-triton

Fast GPTQ kernels written in Triton

gpu-fit-llm

Helps estimate the memory requirements for a model on a GPU.

groqflow

GroqFlow toolchain library

gswarm

Distributed GPU cluster management system with profiling and model orchestration

gtxstorage

GTXStorage Python Library for Amazon S3 Compatible Cloud Storage for Python

happy-vllm

happy_vllm is a REST API for vLLM, production ready

haystack-opea

An integration package connecting OPEA and Haystack

hive-vllm

a

hyperpod-elastic-agent
inferencebench-llm

LLM inference plugin for InferenceBench Suite (vLLM in Phase 1; SGLang/TRT-LLM/llama.cpp/MLX in Phase 2+)

inferiallm

InferiaLLM: distributed inference, filtration, and compute orchestration platform

installm

One-command deployment of OpenAI-compatible APIs for open-source LLMs

instantllm

InstantLLM is the backend server for the free Instant LLM app, enabling users to effortlessly connect and interact with any self-hosted large language model through a user-friendly mobile interface anywhere in the world.

isagellm

sageLLM: Modular LLM inference engine with PD separation for domestic computing power

isagellm-backend

sageLLM backend provider abstraction (CPU/CUDA/Ascend/Kunlun/DCU/MThreads)

isagellm-comm

Communication Layer for sageLLM distributed inference

isagellm-compression

Model Compression & Acceleration Module for sageLLM

isagellm-control-plane

sageLLM Control Plane - Intelligent request routing, scheduling, and engine lifecycle management

isagellm-core

sageLLM core runtime with PD separation (MVP)

isagellm-kv-cache

KV Cache Management Module for sageLLM

isagellm-protocol

sageLLM protocol types and validation (Protocol v0.1)

kani-ext-vllm

vLLM backend for kani

kraken-gstorage

Kraken gstorage

kserve-storage

KServe Storage Handler. This module is responsible to download the models from the provided source

lamina-llm-serve

Centralized model caching and serving layer for Lamina OS

langchain-baseten

Integration package connecting Baseten and LangChain

langchain-kserve

LangChain integration for KServe inference services

langchain-lilypad

The module defines a custom chat model that interacts with the Lilypad API to generate conversational responses and bind tools for enhanced functionality, with mechanisms for parsing tool calls and customizing message payloads.

langchain-llama-server

LangChain integrations for llama-server inference

langchain-llama-stack

An integration package connecting Llama Stack and LangChain

langchain-nvidia-trt

An integration package connecting TritonTensorRT and LangChain

langchain-runpod

LangChain integration package for RunPod Serverless AI endpoints.

langchain-runpod-llm

🐍 | Python library for langchain using RunPod API endpoint as LLM.

langchain-sambanova

An integration package connecting SambaNova and LangChain

langchain-sap-llm
langchain-sarvam

An integration package connecting Sarvam AI and LangChain

langchain-sarvam-integration

Integration of Sarvam AI platform with LangChain

langchain-sarvam-nishant

An integration package connecting Sarvam AI and LangChain.

langextract-vllm

LangExtract provider plugin for VLLM

lazyllm-lmdeploy

A toolset for compressing, deploying and serving LLM

light-llm-hp

A lightweight LLM inference framework

lightllm

lightllm for inference LLM

lit-ollama

Replace ollama with LitServe

llama-api-server

A OpenAI API compatible REST server for llama.

llama-cpp-bin

llama.cpp server binary built from source

llama-cpp-server-py

Add your description here

llama-cpp-server-py-core

Add your description here

llama-deploy-appserver

Application server components for LlamaDeploy

llama-index-llms-heroku

llama-index llms heroku managed inference integration

llama-index-llms-huggingface

llama-index llms huggingface integration

llama-index-llms-huggingface-api

llama-index llms huggingface api integration

llama-index-llms-ibm

llama-index llms IBM watsonx.ai integration

llama-index-llms-konko

llama-index llms konko integration

llama-index-llms-llama-api

llama-index llms llama api integration

llama-index-llms-maritalk

llama-index llms maritalk integration

llama-index-llms-meta

llama-index llms meta llama integration

llama-index-llms-minimax

llama-index llms minimax integration

llama-index-llms-mistral-rs

llama-index llms mistral-rs integration

llama-index-llms-mistralai

llama-index llms mistral ai integration

llama-index-llms-modelscope

llama-index llms modelscope integration

llama-index-llms-modelslab

llama-index llms modelslab integration

llama-index-llms-monsterapi

llama-index llms monsterapi integration

llama-index-llms-mymagic

llama-index llms mymagic integration

llama-index-llms-nebius

llama-index llms Nebius AI Studio integration

llama-index-llms-netmind

llama-index llms netmind integration

llama-index-llms-neutrino

llama-index llms neutrino integration

llama-index-llms-novita

llama-index llms novita integration

llama-index-llms-nvidia

llama-index llms nvidia api catalog integration

llama-index-llms-nvidia-tensorrt

llama-index llms nvidia tensorrt integration

llama-index-llms-nvidia-triton

llama-index llms nvidia triton integration

llama-index-llms-oci-data-science

llama-index llms OCI Data Science integration

llama-index-llms-oci-genai

llama-index llms OCI GenAI integration

llama-index-llms-octoai

llama-index llms octoai integration

llama-index-llms-opea

llama-index llms opea integration

llama-index-llms-openai

llama-index llms openai integration

llama-index-llms-openllm

llama-index llms openllm integration

llama-index-llms-openvino

llama-index llms openvino integration

llama-index-llms-openvino-genai

llama-index llms openvino integration

llama-index-llms-optimum-intel

llama-index llms optimum intel integration

llama-index-llms-sagemaker-endpoint

llama-index llms sagemaker endpoint integration

llama-index-llms-sglang

llama-index llms sglang integration

llama-index-llms-text-generation-inference

llama-index llms huggingface text generation inference integration

llama-index-llms-vllm

llama-index llms vllm integration

llama-index-llms-xinference

llama-index llms xinference integration

llama-optimus

llama-optimus is a lightweight Python tool to automatically optimize llama.cpp performance flags for maximum tg & pp token/s throughput. Powered by Bayesian optimization with Optuna

llama-slobber

Slobr utilities

llama-stack

Open-source, OpenAI-compatible API server with pluggable providers for any model and any infrastructure

llama-stack-api

API and Provider specifications for Llama Stack - lightweight package with protocol definitions and provider specs

llama_cpp_http

Python llama.cpp HTTP Server and LangChain LLM Client

llamacpp-client

Client for LlamaCpp HTTP Server

llamactl

A command-line interface for managing LlamaDeploy projects and deployments

llamajuice

Squeeze every last drop from your NVIDIA GPU.

More in Agent frameworks & infrastructure