Agent frameworks & infrastructure
Model serving & inference
775 agents · most starred first
Model serving & inference, page 3 of 8
AWS SDK for JavaScript Sagemaker Client for Node.js, Browser and React Native
The official TypeScript library for the Cerebras API
Inference API for Azure-supported AI models
AWS SDK for JavaScript Sagemaker Runtime Client for Node.js, Browser and React Native
JavaScript and TypeScript client for Gradient AI
Vertex AI client for Node.js
Multi-agent inference inside the decode loop — structured concurrency over shared KV state
GEIANT Hive provider for OpenClaw — cryptographically-signed distributed local AI inference with EU AI Act compliant agent identity
Production ready LLM model compression/quantization toolkit with hw accelerated inference support for both cpu/gpu via HF, vLLM, and SGLang.
This library is the preferred way of accessing Google Cloud Storage from App Engine. It was designed to replace the Files API. As a result it contains much of the same functionality (streaming reads and writes but not the complete set of GCS APIs). It also provides key stability improvements and a better overall developer experience.
This library is the preferred way of accessing Google
Ablation plugin for vLLM
VLLM performance testing actuator for ado
Python bindings for the AeroLLM runtime — streaming inference for LLMs that don't fit in GPU memory
ag_llama_api Test Package for Somthing
ag_llama_api_s Test Package for Somthing
ag_llama_hub Test Package for Somthing
ag_llama_hub Test Package for Somthing
ag_llama2_api_s Test Package for Somthing
ag_llama2_hub Test Package for Somthing
A Python package for agent inference capabilities
A Python package for agentic inference capabilities
A high-throughput and memory-efficient inference and serving engine for LLMs
An integration of AWS S3 and Bedrock as a Downloader and Generator components.
An integration of Amazon Sagemaker as an SagemakerGenerator component.
The official Python library for the anthropic-bedrock API
An easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.
Official Python bindings for AutoAgents mistral-rs backend (CUDA)
Official Python bindings for AutoAgents mistral-rs backend (Metal)
Official Python bindings for AutoAgents mistral-rs backend (CPU)
Pydantic v2 models for AWS bedrock_runtime, shipped as a PEP 420 namespace extension of aws-resource-validator.
CDK constructs for defining an interaction between an AWS Lambda function and an Amazon Bedrock inference profile.
Python client for Amazon Bedrock Runtime
Azure Storage Connector for PyTorch
Azure Blob Storage as object cache
Library for running inference on large language models with the ability to remove generated tokens.
OpenAI's response format for its open-weight model series gpt-oss
A framework designed for llm offline distributed inference from Odps datasource.
BetterAirLLM allows single 4GB GPU card to run 70B large language models without quantization, distillation or pruning. 8GB vmem to run 405B Llama3.1.
Simple paged attention with KV cache for RL scenarios
Dataclass for boto3 bedrock.
Dataclass for boto3 bedrock_agent.
Dataclass for boto3 bedrock_agent_runtime.
Dataclass for boto3 bedrock_agentcore.
Dataclass for boto3 bedrock_agentcore_control.
Dataclass for boto3 bedrock_runtime.
Dataclass for boto3 kinesis_video_webrtc_storage.
Dataclass for boto3 storagegateway.
backupstorage data for botocore. See the `botocore-a-la-carte` package for more info.
bedrock data for botocore. See the `botocore-a-la-carte` package for more info.
bedrock-agent data for botocore. See the `botocore-a-la-carte` package for more info.
bedrock-agent-runtime data for botocore. See the `botocore-a-la-carte` package for more info.
bedrock-runtime data for botocore. See the `botocore-a-la-carte` package for more info.
kinesis-video-webrtc-storage data for botocore. See the `botocore-a-la-carte` package for more info.
storagegateway data for botocore. See the `botocore-a-la-carte` package for more info.
VLLM adapters for Bridgic, enabling connectivity with self-hosted LLM providers.
This is a reserved placeholder package to mitigate dependency confusion risks.
This is a reserved placeholder package to mitigate dependency confusion risks.
This is a reserved placeholder package to mitigate dependency confusion risks.
This is a reserved placeholder package to mitigate dependency confusion risks.
ByzerLLM: Byzer LLM
A tool that allows you to use the OpenAI API with vLLM generate function
@cdklabs/genai-idp-bedrock-llm-processor
A Python library for the Cohere endpoints in AWS Sagemaker & Bedrock
A Python library for the Cohere endpoints in AWS Sagemaker
KoalaVault Key Provider for CryptoTensors - Secure key management for encrypted model deployment with vLLM
High-performance GPT implementation using NVIDIA CUDA Tile Programming
Add your description here
Add your description here
Add your description here
High-performance inference engine for Diffusion Language Models - 3x faster with advanced optimizations
vLLM for DGX Spark (GB10). After pip install, run: sudo dgx-spark-vllm-install
Distributed Inference for Large Language Models.
Dora OpenAI API Server
Hardware-software co-design framework for AI accelerators
Fast GPU tensor loading via DirectStorage on Windows (NVMe -> GPU, no CPU copy)
Accelerating LLMs by 3x with No Quality Loss
Easy to use lightweight vllm tool with special support for Large Reasoning Model (LRM).
Single-user optimized inference wrapper for ExLlamaV3
extend for vllm
FastDeploy for Large Language Model
boost inference speed of GPT models in transformers by onnxruntime
Reusable LLM clients and speculative decode workflows for FlexInfer services
vLLM plugin for flyte
High-performance C++ inference engine for dense and MoE language models
Fastllm
Fastllm
Furiosa LLM
Furiosa LLM
An opinionated Llama Server engine with a focus on agentic tasks
GenZ is designed to simplify the relationship between the hardware platform used for serving Large Language Models(LLMs) and inference serving metrics like latency and memory.
Python LLM operations service for the DevLama ecosystem
A library containing components related to model inferences in Gen AI applications.
A high-throughput and memory-efficient inference and serving engine for LLMs