Agent frameworks & infrastructure
Model serving & inference
775 agents · most starred first
Model serving & inference, page 4 of 8
A collection of reference inference implementations for gpt-oss by OpenAI
API client for GPT-2 text generator cloud hosted by Open Medical IO
Easy-to-use Wrapper for the GPT-2 117M, 345M, 774M, and 1.5B Transformer Models
Accelerate transformer inference by 7.6-9x. Native to Huggingface and PyTorch.
A GPT-J api to use with python3 to generate text, blogs, code, and more (Note: Starting with version 3.0.7 the api is using the old domain again so there might be some issues with limits)
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
A Python package for GPTQ quantization
OpenAI compatible API server for AutoGPTQ model
Fast GPTQ kernels written in Triton
Helps estimate the memory requirements for a model on a GPU.
GroqFlow toolchain library
Distributed GPU cluster management system with profiling and model orchestration
GTXStorage Python Library for Amazon S3 Compatible Cloud Storage for Python
happy_vllm is a REST API for vLLM, production ready
An integration package connecting OPEA and Haystack
a
LLM inference plugin for InferenceBench Suite (vLLM in Phase 1; SGLang/TRT-LLM/llama.cpp/MLX in Phase 2+)
InferiaLLM: distributed inference, filtration, and compute orchestration platform
One-command deployment of OpenAI-compatible APIs for open-source LLMs
InstantLLM is the backend server for the free Instant LLM app, enabling users to effortlessly connect and interact with any self-hosted large language model through a user-friendly mobile interface anywhere in the world.
sageLLM: Modular LLM inference engine with PD separation for domestic computing power
sageLLM backend provider abstraction (CPU/CUDA/Ascend/Kunlun/DCU/MThreads)
Communication Layer for sageLLM distributed inference
Model Compression & Acceleration Module for sageLLM
sageLLM Control Plane - Intelligent request routing, scheduling, and engine lifecycle management
sageLLM core runtime with PD separation (MVP)
KV Cache Management Module for sageLLM
sageLLM protocol types and validation (Protocol v0.1)
vLLM backend for kani
Kraken gstorage
KServe Storage Handler. This module is responsible to download the models from the provided source
Centralized model caching and serving layer for Lamina OS
Integration package connecting Baseten and LangChain
LangChain integration for KServe inference services
The module defines a custom chat model that interacts with the Lilypad API to generate conversational responses and bind tools for enhanced functionality, with mechanisms for parsing tool calls and customizing message payloads.
LangChain integrations for llama-server inference
An integration package connecting Llama Stack and LangChain
An integration package connecting TritonTensorRT and LangChain
LangChain integration package for RunPod Serverless AI endpoints.
🐍 | Python library for langchain using RunPod API endpoint as LLM.
An integration package connecting SambaNova and LangChain
An integration package connecting Sarvam AI and LangChain
Integration of Sarvam AI platform with LangChain
An integration package connecting Sarvam AI and LangChain.
LangExtract provider plugin for VLLM
A toolset for compressing, deploying and serving LLM
A lightweight LLM inference framework
lightllm for inference LLM
Replace ollama with LitServe
A OpenAI API compatible REST server for llama.
llama.cpp server binary built from source
Add your description here
Add your description here
Application server components for LlamaDeploy
llama-index llms heroku managed inference integration
llama-index llms huggingface integration
llama-index llms huggingface api integration
llama-index llms IBM watsonx.ai integration
llama-index llms konko integration
llama-index llms llama api integration
llama-index llms maritalk integration
llama-index llms meta llama integration
llama-index llms minimax integration
llama-index llms mistral-rs integration
llama-index llms mistral ai integration
llama-index llms modelscope integration
llama-index llms modelslab integration
llama-index llms monsterapi integration
llama-index llms mymagic integration
llama-index llms Nebius AI Studio integration
llama-index llms netmind integration
llama-index llms neutrino integration
llama-index llms novita integration
llama-index llms nvidia api catalog integration
llama-index llms nvidia tensorrt integration
llama-index llms nvidia triton integration
llama-index llms OCI Data Science integration
llama-index llms OCI GenAI integration
llama-index llms octoai integration
llama-index llms opea integration
llama-index llms openai integration
llama-index llms openllm integration
llama-index llms openvino integration
llama-index llms openvino integration
llama-index llms optimum intel integration
llama-index llms sagemaker endpoint integration
llama-index llms sglang integration
llama-index llms huggingface text generation inference integration
llama-index llms vllm integration
llama-index llms xinference integration
llama-optimus is a lightweight Python tool to automatically optimize llama.cpp performance flags for maximum tg & pp token/s throughput. Powered by Bayesian optimization with Optuna
Slobr utilities
Open-source, OpenAI-compatible API server with pluggable providers for any model and any infrastructure
API and Provider specifications for Llama Stack - lightweight package with protocol definitions and provider specs
Python llama.cpp HTTP Server and LangChain LLM Client
Client for LlamaCpp HTTP Server
A command-line interface for managing LlamaDeploy projects and deployments
Squeeze every last drop from your NVIDIA GPU.