Inference agents

A high-throughput and memory-efficient inference and serving engine for LLMs

Port of OpenAI's Whisper model in C/C++

A generative speech model for daily dialogue.

Langchain-Chatchat（原Langchain-ChatGLM）基于 Langchain 与 ChatGLM, Qwen 与 Llama 等语言模型的 RAG 与 Agent 应用 |…

Faster Whisper transcription with CTranslate2

Multi-lingual large voice generation model, providing inference, training and deployment full-stack ability.

A list of free LLM inference resources accessible via API.

Welcome to the Llama Cookbook! This is your go to guide for Building with Llama: Getting started with…

🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading

Plano is an AI-native proxy and data plane for agentic apps — with built-in orchestration, safety…

A high-performance LLM inference API and Chat UI that integrates DeepSeek R1's CoT reasoning traces with…

Superduper: End-to-end framework for building custom AI applications and agents.

Eko (Eko Keeps Operating) - Build Production-ready Agentic Workflow with Natural Language - eko.fellou.ai

A GPU cluster manager that configures and orchestrates inference engines like vLLM and SGLang for…

CSGHub is a brand-new open-source platform for managing LLMs, developed by the OpenCSG team. It offers both…

Low-latency AI engine for mobile devices & wearables

FEDML - The unified and scalable ML library for large-scale distributed training, model serving, and…

Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.

High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle

Optimizing inference proxy for LLMs

A portable accelerated SQL query, search, and LLM-inference engine, written in Rust, for data-grounded AI…

A library of enterprise-grade AI agents designed to democratize artificial intelligence and provide free…

⚡ Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for LLMs; run…

14-stage Fusion Pipeline for LLM token compression — reversible compression, AST-aware code analysis…

Control plane for agents and engineers to provision compute and run training and inference across NVIDIA…

Run any Llama 2 locally with gradio UI on GPU or CPU from anywhere (Linux/Windows/Mac). Use `llama2-wrapper`…

The PHP Agentic Framework to build production-ready AI driven applications. Connect components (LLMs, vector…

[ICML 2024] LLMCompiler: An LLM Compiler for Parallel Function Calling

A beautiful local-first coding agent running in your terminal - built by the community for the community ⚒

An agentic company research tool powered by LangGraph and Tavily that conducts deep diligence on companies…

Build Anything with AI Agents

This course is designed to guide beginners through the exciting world of Edge AI, covering fundamental…

Offline inference engine for art, real-time voice conversations, LLM powered chatbots and automated workflows

Jlama is a modern LLM inference engine for Java

List of software that allows searching the web with the assistance of AI…

Parallax is a distributed model serving framework that lets you build your own AI cluster anywhere

Highly Performant, Modular, Memory Safe and Production-ready Inference, Ingestion and Indexing built in Rust 🦀

Route, manage, and analyze your LLM requests across multiple providers with a unified API interface.

OpenAlpha_Evolve is an open-source Python framework inspired by the groundbreaking research on autonomous…

Running Llama 2 and other Open-Source LLMs on CPU Inference Locally for Document Q&A

OpenTelemetry Instrumentation for AI Observability

OpenAI and Anthropic compatible server for Apple Silicon. Run LLMs and vision-language models (Llama…

Llama 3+ inference in pure Java

Browser-LLM Auto-Scaling Technology

One API for all LLMs either Private or Public (Anthropic, Llama V2, GPT 3.5/4, Vertex, GPT4ALL, HuggingFace…

MLX Omni Server is a local inference server powered by Apple's MLX framework, specifically designed for Apple…

Run LLaMA (and Stanford-Alpaca) inference on Apple Silicon GPUs.

CLI for running large numbers of coding agents in parallel with git worktrees

Minimalist web-searching platform with an AI assistant that runs directly from your browser. Uses WebLLM…

🏗️ Fine-tune, build, and deploy open-source LLMs easily!

Ollama alternative for Rockchip NPU: An efficient solution for running AI and Deep learning models on…

irresponsible innovation. Try now at https://chat.dev/

A command-line interface tool for serving LLM using vLLM.

✨ AI interface for tinkerers (Ollama, Haystack RAG, Python)

A Framework for LLM-based Multi-Agent Reinforced Training and Inference

Local AI anywhere, for everyone — LLM inference, chat UI, voice, agents, workflows, RAG, and image…

Design, conduct and analyze results of AI-powered surveys and experiments. Simulate social science and market…

LLMs and Machine Learning done easily

MLX Studio - Home of JANG_Q - Image Gen/Edit + Chat/Code All in one - + OpenClaw (Anthropic API)

A tool for generating function arguments and choosing what function to call with local LLMs

Super performant RAG pipelines for AI apps. Summarization, Retrieve/Rerank and Code Interpreters in one…

Inference engine for Intel devices. Serve LLMs, VLMs, Whisper, Kokoro-TTS, Embedding and Rerank models over…

Context-Engine MCP - Agentic Context Compression Suite

Optimized local inference for LLMs with HuggingFace-like APIs for quantization, vision/language models…

Explore the unknown, build the future, own your data.

AI-powered cybersecurity chatbot designed to provide helpful and accurate answers to your…

inference.sh Agent skills for using our API to give your agents access to hundreds of apps and other agents

💬 Chatbot web app + HTTP and Websocket endpoints for LLM inference with the Petals client

Zig INferenCe Engine — Local LLM inference on AMD GPUs and Apple Silicon

Local AI Assistant on your phone

A collection of Kotlin-based examples featuring AI frameworks such as Spring AI, LangChain4j, and more …

A simple, fast and robust program-aware agentic inference system.

Music Analysis, Chord Recognition, Beat Tracking, Guitar Diagrams, Piano Visualizer, Lyrics Transcription…

Mano-P: Open-source GUI-VLA agent for edge devices. #1 on OSWorld (specialized, 58.2%). Runs locally on Apple…

DePIN for Vintage Hardware — Proof-of-Antiquity blockchain where old machines outmine new ones. AI-powered…

Bespoke Automata is a GUI and deployment pipline for making complex AI agents locally and offline

🏆 gym-cooking: Code for "Too many cooks: Bayesian inference for coordinating multi-agent collaboration"…

OpenAI compatible API for TensorRT LLM triton backend

PocketGroq is a powerful Python library that simplifies integration with the Groq API, offering advanced…

A production-ready platform for dynamic AI agents — plan, use tools, and complete real work without hardcoded…

Open-source, fully private and local alternative to NotebookLM. Chat with your documents, generate audio…

Graphsignal Python SDK

A template to create any LLM Inference Web Apps using Python only

Decapod is the daemonless, local-first control plane agents call on demand to converge on human intent, shape…

Kubernetes enhancements for Network Topology Aware Gang Scheduling & Autoscaling

HiveMind Protocol - A Local-First, Privacy-Preserving Architecture for Agentic RAG

Context Encoding for Semantic Segmentation MegaDepth: Learning Single-View Depth Prediction from Internet…

An implementation of Google Deep Search 🕵️ with support for 1000+ references, local inference, chatting with…

[arxiv: 2503.23895] Dynamic Parametric Retrieval Augmented Generation for Test-time Knowledge Enhancement

Run any Large Language Model behind a unified API

Booster - open accelerator for LLM models. Better inference and debugging for AI hackers

🦙 Free and Open Source Large Language Model (LLM) chatbot web UI and API. Self-hosted, offline capable and…

Engine-agnostic LLM gateway in Rust. Full OpenAI & Anthropic API compatibility across SGLang, vLLM, TRT-LLM…

Higher performance OpenAI LLM service than vLLM serve: A pure C++ high-performance OpenAI LLM service…

Turn your company's scattered knowledge into AI ready Books ✨

kongruity leverages LLMs and cohesion scoring to transform unstructured creative-engineering process…

Multi-Agent Conversation Framework in TypeScript

🪶 Lightweight OpenAI drop-in replacement for Kubernetes

Openai-style, fast & lightweight local language model inference w/ documents

EcoAssistant: using LLM assistant more affordably and accurately

开源的智能体项目 支持6种聊天平台 Onebotv11一对多连接 流式信息 agent 对话keyboard气泡生成 支持10+大模型接口(持续更新) 具有将多种大模型接口转化为带有上下文的通用格式的能力.

GenAI & agent toolkit for Apple Silicon Mac, implementing JSON schema-steered structured output (3SO) and…

A blazingly fast, privacy first & OPEN AI Chat Interface

Large Language Model (LLM) Inference API and Chatbot

Run large language models like Qwen and LLaMA locally on Android for offline, private, real-time question…

A curated list of tools, papers, and datasets for applying AI to cybersecurity tasks. This list primarily…

A simple NPM interface for seamlessly interacting with 36 Large Language Model (LLM) providers, including…

SLED: Self Logits Evolution Decoding for Improving Factuality in Large Language Model …

Designing Multi-Agent Systems with Zero Supervision

Run a <400ms latency Voice Agent on just 4GB VRAM. Fully offline, no API keys required. Optimized for GTX…

Using Large Language Models for Repo-wide Type Prediction

OpenAI-compatible HTTP LLM proxy / gateway for multi-provider inference (Google, Anthropic, OpenAI, PyTorch)…

All the code and materials

An open-source, cloud-native, high-performance gateway unifying multiple LLM providers, from local solutions…

Modern desktop application (Rust + Tauri v2 + Svelte 5 + Candle (HF)) for communicating with AI models that…

BlazorGPT is a Blazor Server application that uses Semantic Kernel plus OpenAI, Azure OpenAI and Ollama for…

Deep active inference agents using Monte-Carlo methods

True on-device AI for Kotlin Multiplatform (Android, iOS, Desktop, JVM, WASM). LLM, Speech-to-Text and Image…

Gradio based tool to run opensource LLM models directly from Huggingface

Langport is a language model inference service

Open source subtitling platform 💻 for transcribing and translating videos/audios in Indic languages.

Extensible generative AI platform on Kubernetes with OpenAI-compatible APIs.

Orchestrator Kit for Agentic Reasoning - OrKa is a modular AI orchestration system that transforms Large…

Enterprise-grade memory framework for LLMs featuring GPU-optimized inference, vector storage, and automated…

Route inference across LLM providers. Track cost per request.

On-device AI for iOS & Android

🦑 Unified Swift SDK for LLM inference across local and cloud providers

Multimodal AI agent, an interactive data studio with on-demand ML inference, media generation, and a database…

A declarative way to control LLMs.

PasLLM - LLM inference engine in Object Pascal (synced from my private work repository)

A rationalist ruleset for "debugging" LLMs, auditing their internal reasoning and uncovering biases; also a…

On-device AI SDK for Flutter — LLM inference, vision, STT, TTS, image generation, embeddings, RAG, and…

Temporal Code Intelligence platform. Time-series complexity analysis across Python, JavaScript, Java, and Go…

Confidential is software for private, secure AI workloads, agents, and inference. It lets you provide on-prem…

Accelerating Long Context LLM Inference with Accuracy-Preserving Context Optimization in SGLang, vLLM…

Monocle is a framework for tracing GenAI app code. This repo contains implementation of Monocle for GenAI…

Inference Hub for AI at Scale

Flowchart-like UI to interconnect LLM's and Huggingface models, and deploy them as a REST API with little to…

Build, deploy, and orchestrate event-driven agents natively on Apache Flink® and Apache Kafka®

This project applies the core knowledge from the LLMOps module, including the design and implementation of…

It is a comprehensive resource hub compiling all LLM papers accepted at the International Conference on…

[SOICT 2024] LLM-Powered Video Search: A Comprehensive Multimedia Retrieval System

LLM inference in Fortran

Build an Autonomous Web3 AI Trading Agent (BASE + Uniswap V4 example)

A curated, hands-on library of notebooks, demos, and resources for AI/ML, Deep Learning, Generative AI…

Stop forcing LLMs to answer in one pass. Give them a runtime. Recursive Language Model that improves any LLM…

The most feature-complete local AI workstation. Multi-GPU inference, integrated Stable Diffusion + ADetailer…

A tiny LLM Agent with minimal dependencies, focused on local inference.

152 open-source tools to run LLMs 100% locally – no cloud, no API keys, no censorship

Cross-Platform High-Level LLM Library

AI assistant for Home Assistant with cognitive memory. Supports Anthropic, OpenAI, and Ollama (local…

Extract structured data from local or remote LLM models

run ollama & gguf easily with a single command

PHP library for interacting with AI platform provider.

A sovereign cognitive architecture with IIT 4.0 integrated information, residual-stream affective steering…

Multi-core, Tokio-native orchestration for LLM pipelines.

A comprehensive toolkit for deploying production-ready Generative AI infrastructure on Amazon EKS. Includes…

Biologically-inspired persistent memory engine for Claude Code. 26 cognitive subsystems, Hopfield networks…

A distributed system for Agentic AI

Browser based Interface for Generative AI. Chat/Agent/Taskmanager Hybrid.

Neuro-Symbolic-Causal AI - Project Chimera | 🌌 An open research project exploring formal verification of AI…

Testing speed and accuracy of RAG with, and without Cross Encoder Reranker.

Pull high-quality, efficient embeddings for PubMed, arXiv and Wikipedia from Huggingface and use for local…

PyTorch implementation of the paper: Multi-Agent Collaborative Inference via DNN Decoupling: Intermediate…

Zero-code LLM security & observability proxy. Real-time prompt injection detection, PII scanning, and cost…

The complete AI platform on a $3 microcontroller. Sub-millisecond inference. Zero hallucinations.

Python package to create adversarial agents for membership inference attacks againts machine learning models

llmBench is a high-depth benchmarking tool designed to measure the raw performance of local LLM runtimes…

Wingman is the fastest and easiest way to run Llama models on your PC or Mac.

[NeurIPS 2025 𝐒𝐩𝐨𝐭𝐥𝐢𝐠𝐡𝐭] AutoToM: Scaling Model-based Mental Inference via Automated Agent Modeling

BlockRank makes LLMs efficient and scalable for RAG and in-context ranking

Training and inference code for "Training Language Models for Social Deduction with Multi-Agent Reinforcement…

Helios Engine is a powerful and flexible Rust framework for building LLM-powered agents with tool support…

Qurio brings multi-provider models, custom agents, reusable skills, MCP servers, HTTP tools, retrieval…

OpenVitamin is a local-first AI execution platform that unifies Agents, Workflows, and multi-model inference…

MOTO - Autonomous ASI Deep Research Harness by Intrafere - creative novelty-seeking researcher for S.T.E.M…

Android native AI inference library, bringing gguf models and stable-diffusion inference on android devices…

Empower Your Productivity with Local AI Assistants

LM-Kit Maestro is a secure, innovative desktop application that orchestrates AI agents offline, empowering…

ht - a shell command that answers your questions about shell commands

A Framework for Narrative Agents

CrewNews is an AI news generator that delivers an unbiased version of the news for a given topic, using…

A proxy server that intercepts Anthropic API requests and converts them to OpenAI-compatible format, enabling…

Live2D + ASR + LLM + TTS → Real-time communication + Offline Deployment/Cloud Inference 实时沟通 本地部署/云端推理

An end-to-end pipeline to optimize and host LLM for 100K parallel queries

axe - a precision agentic coder. large codebases. zero bloat. terminal-native. precise retrieval. powerful…

这是一个由LangGraph协议主导的因果分析Muti-Agent，结合MCP,RAG等多种工具进行辅助进行因果分析，提供给用户一份完善的因果分析的分析报告和因果图

Production inference for encoder models - ColBERT, GLiNER, ColPali, embeddings etc. - as vLLM plugins for…

🔥 🔥 Alternative to Ollama 🔥 🔥 multi-model <1ms LLM switching

Financial CrewAI Agents (LangChain, YF Tools, Ai Crew, Groq Inference)

the small distributed language model toolkit; fine-tune state-of-the-art LLMs anywhere, rapidly

Complete observability platform for AI agents and MCP servers. Improve your AI deployment outcomes, identify…

Fluent GPT App, an open-source, multi-platform desktop application that brings the power of GPT models to…

A Python framework for building AI agent systems with robust task management in the form of a graph execution…

Native and Private ML inference engine, embeddings, classification, reranking, search, and text generation…

[KDD 2023] Ball Trajectory Inference from Multi-Agent Sports Contexts Using Set Transformer and Hierarchical…

AI agent with multi-agent orchestration, autonomous cognitive systems, and a full management dashboard

A local browser automation agent based on Microsoft Fara-7B model optimized for LM Studio inference.

Fast GPU OCR server. 270 img/s on FUNSD. TensorRT FP16, PP-OCRv5, HTTP + gRPC.