Agent frameworks & infrastructure

Model serving & inference

775 agents · most starred first

Model serving & inference, page 3 of 8

@aws-sdk/client-sagemaker

AWS SDK for JavaScript Sagemaker Client for Node.js, Browser and React Native

@cerebras/cerebras_cloud_sdk

The official TypeScript library for the Cerebras API

@azure-rest/ai-inference

Inference API for Azure-supported AI models

@aws-sdk/client-sagemaker-runtime

AWS SDK for JavaScript Sagemaker Runtime Client for Node.js, Browser and React Native

@gradientai/nodejs-sdk

JavaScript and TypeScript client for Gradient AI

@google-cloud/aiplatform

Vertex AI client for Node.js

@lloyal-labs/lloyal-agents

Multi-agent inference inside the decode loop — structured concurrency over shared KV state

@gns-foundation/openclaw-hive-provider

GEIANT Hive provider for OpenClaw — cryptographically-signed distributed local AI inference with EU AI Act compliant agent identity

GPTQModel

Production ready LLM model compression/quantization toolkit with hw accelerated inference support for both cpu/gpu via HF, vLLM, and SGLang.

GoogleAppEngineCloudStorageClient

This library is the preferred way of accessing Google Cloud Storage from App Engine. It was designed to replace the Files API. As a result it contains much of the same functionality (streaming reads and writes but not the complete set of GCS APIs). It also provides key stability improvements and a better overall developer experience.

GoogleAppEngineCloudStorageClientPy3

This library is the preferred way of accessing Google

ablated-vllm-plugin

Ablation plugin for vLLM

ado-vllm-performance

VLLM performance testing actuator for ado

aerollm-api

Python bindings for the AeroLLM runtime — streaming inference for LLMs that don't fit in GPU memory

ag-llama-api

ag_llama_api Test Package for Somthing

ag-llama-api-s

ag_llama_api_s Test Package for Somthing

ag-llama-hub

ag_llama_hub Test Package for Somthing

ag-llama-hub-s

ag_llama_hub Test Package for Somthing

ag-llama2-api-s

ag_llama2_api_s Test Package for Somthing

ag-llama2-hub-s

ag_llama2_hub Test Package for Somthing

agent-inference

A Python package for agent inference capabilities

agentic-inference

A Python package for agentic inference capabilities

agentopera
ai-dynamo-vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

amazon-bedrock-haystack

An integration of AWS S3 and Bedrock as a Downloader and Generator components.

amazon-sagemaker-haystack

An integration of Amazon Sagemaker as an SagemakerGenerator component.

anthropic-bedrock

The official Python library for the anthropic-bedrock API

auto-gptq

An easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.

autoagents-mistral-rs-cuda

Official Python bindings for AutoAgents mistral-rs backend (CUDA)

autoagents-mistral-rs-metal

Official Python bindings for AutoAgents mistral-rs backend (Metal)

autoagents-mistral-rs-py

Official Python bindings for AutoAgents mistral-rs backend (CPU)

aws-resource-validator-bedrock-runtime

Pydantic v2 models for AWS bedrock_runtime, shipped as a PEP 420 namespace extension of aws-resource-validator.

aws-solutions-constructs.aws-lambda-bedrockinferenceprofile

CDK constructs for defining an interaction between an AWS Lambda function and an Amazon Bedrock inference profile.

aws_sdk_bedrock_runtime

Python client for Amazon Bedrock Runtime

azstoragetorch

Azure Storage Connector for PyTorch

azure-blob-storage-cache

Azure Blob Storage as object cache

backtracking-llm

Library for running inference on large language models with the ability to remove generated tokens.

baseten-openai-harmony

OpenAI's response format for its open-weight model series gpt-oss

batchllm

A framework designed for llm offline distributed inference from Odps datasource.

betterairllm

BetterAirLLM allows single 4GB GPU card to run 70B large language models without quantization, distillation or pruning. 8GB vmem to run 405B Llama3.1.

bigdl-core-llm
bllm-inference

Simple paged attention with KV cache for RL scenarios

boto3-dataclass-bedrock

Dataclass for boto3 bedrock.

boto3-dataclass-bedrock-agent

Dataclass for boto3 bedrock_agent.

boto3-dataclass-bedrock-agent-runtime

Dataclass for boto3 bedrock_agent_runtime.

boto3-dataclass-bedrock-agentcore

Dataclass for boto3 bedrock_agentcore.

boto3-dataclass-bedrock-agentcore-control

Dataclass for boto3 bedrock_agentcore_control.

boto3-dataclass-bedrock-runtime

Dataclass for boto3 bedrock_runtime.

boto3-dataclass-kinesis-video-webrtc-storage

Dataclass for boto3 kinesis_video_webrtc_storage.

boto3-dataclass-storagegateway

Dataclass for boto3 storagegateway.

botocore-a-la-carte-backupstorage

backupstorage data for botocore. See the `botocore-a-la-carte` package for more info.

botocore-a-la-carte-bedrock

bedrock data for botocore. See the `botocore-a-la-carte` package for more info.

botocore-a-la-carte-bedrock-agent

bedrock-agent data for botocore. See the `botocore-a-la-carte` package for more info.

botocore-a-la-carte-bedrock-agent-runtime

bedrock-agent-runtime data for botocore. See the `botocore-a-la-carte` package for more info.

botocore-a-la-carte-bedrock-runtime

bedrock-runtime data for botocore. See the `botocore-a-la-carte` package for more info.

botocore-a-la-carte-kinesis-video-webrtc-storage

kinesis-video-webrtc-storage data for botocore. See the `botocore-a-la-carte` package for more info.

botocore-a-la-carte-storagegateway

storagegateway data for botocore. See the `botocore-a-la-carte` package for more info.

bridgic-llms-vllm

VLLM adapters for Bridgic, enabling connectivity with self-hosted LLM providers.

bytedance-llm-global-scheduling

This is a reserved placeholder package to mitigate dependency confusion risks.

bytedance-xgpt-kernels

This is a reserved placeholder package to mitigate dependency confusion risks.

bytedance-xgpt-server

This is a reserved placeholder package to mitigate dependency confusion risks.

bytedxperf-gpt-triton

This is a reserved placeholder package to mitigate dependency confusion risks.

byzerllm

ByzerLLM: Byzer LLM

caai-vllm-tools

A tool that allows you to use the OpenAI API with vLLM generate function

cdklabs.genai-idp-bedrock-llm-processor

@cdklabs/genai-idp-bedrock-llm-processor

cohere-aws

A Python library for the Cohere endpoints in AWS Sagemaker & Bedrock

cohere-sagemaker

A Python library for the Cohere endpoints in AWS Sagemaker

cryptotensors-koalavault-vllm

KoalaVault Key Provider for CryptoTensors - Secure key management for encrypted model deployment with vLLM

cutile-gpt

High-performance GPT implementation using NVIDIA CUDA Tile Programming

datadog-vllm
dbgpt-acc-auto

Add your description here

dbgpt-acc-flash-attn

Add your description here

dbgpt-serve

Add your description here

dfastllm

High-performance inference engine for Diffusion Language Models - 3x faster with advanced optimizations

dgx-spark-vllm

vLLM for DGX Spark (GB10). After pip install, run: sudo dgx-spark-vllm-install

distllm

Distributed Inference for Large Language Models.

dora-openai-server

Dora OpenAI API Server

dora-openai-websocket
dragonx-optimizer

Hardware-software co-design framework for AI accelerators

dstorage-gpu

Fast GPU tensor loading via DirectStorage on Windows (NVMe -> GPU, no CPU copy)

eagle-llm

Accelerating LLMs by 3x with No Quality Loss

easyvllm

Easy to use lightweight vllm tool with special support for Large Reasoning Model (LRM).

exllamav3-inference

Single-user optimized inference wrapper for ExLlamaV3

exvllm

extend for vllm

fastdeploy-llm

FastDeploy for Large Language Model

fastgpt

boost inference speed of GPT models in transformers by onnxruntime

flexinfer-llm-kit

Reusable LLM clients and speculative decode workflows for FlexInfer services

flyteplugins-vllm

vLLM plugin for flyte

ftllm

High-performance C++ inference engine for dense and MoE language models

ftllm-nightly

Fastllm

ftllm-rocm

Fastllm

furiosa-llm

Furiosa LLM

furiosa-llm-models

Furiosa LLM

furiosa-llm-native
furiosa-native-llm-common
gallama

An opinionated Llama Server engine with a focus on agentic tasks

genz-llm

GenZ is designed to simplify the relationship between the hardware platform used for serving Large Language Models(LLMs) and inference serving metrics like latency and memory.

getllm

Python LLM operations service for the DevLama ecosystem

gllm-inference-binary

A library containing components related to model inferences in Gen AI applications.

gllm-rt

A high-throughput and memory-efficient inference and serving engine for LLMs

More in Agent frameworks & infrastructure