Agent frameworks & infrastructure
Local LLM runtimes
1,401 agents · most starred first
Local LLM runtimes, page 13 of 15
PetaLLM allows single 4GB GPU card to run 70B large language models without quantization, distillation or pruning. 8GB vmem to run 405B Llama3.1.
picoLLM Inference Engine
picoLLM Inference Engine demos
PIP-BOY LLM - Vault-Tec Local AI Terminal
PixelArray 像素云盘客户端
Manage multiple Ollama servers and fan out parallel inference across them.
Fix ollama pull TLS timeout and disconnects — resumable Ollama model downloader for slow connections
Agent toolkit for Ollama
Add your description here
Run MLX compatible HuggingFace models with Pydantic AI locally
Pydantic AI Ollama integration
Official Python CPU inference for GPT4All language models based on llama.cpp and ggml
Python bindings for the GGML GPT-J Laguage model
pyllama2
Python bindings for llama.cpp
Fast and lightweight multimodal LLM inference engine for mobile and edge devices
LLM extension for Pythonaibrain with quantized model support
Local runner for QuantGPT with WebSocket support
Run Ollama models easily, anywhere – including online platforms like Google Colab
fast startup ollama server
A simple SDK for interacting with the Ollama API by automatically creating a conversation (chat history)
Run 70B+ LLMs on a single 4GB GPU — no quantization required. Layer-streaming inference for consumer hardware.
A simple custom LLM wrapper for llama.cpp with LangChain compatibility.
A platform for reworkd
LLM in python
Simple CLI runner for Ollama (LLM) models
Scikit-Ollama: an extension of Scikit-LLM for Ollama served models.
A package for using self-served LLM
llama cpp integration for serapeum
Ollama integration for Serapeum
Running and training LLMs on Apple Silicon via MLX
VLLM & Unsloth Wrapper for simple local inference.
Sinapsis templates for llama.cpp text generation.
Super Lazy Language Model
Shared Ollama gateway over Slurm — wake a GPU node on demand
A package that enhances local LLMs with real-time web awareness and adaptive context retrieval, making them smarter with no retraining or fine-tuning.
A python app to serve smOllama
A Streamlit chatbot app integrating Ollama LLMs
Ollama provider for Stache AI
Karpathy's pure-Python microGPT as a Strands Agents model provider — zero dependencies, pure autograd, from scratch.
Streamlit utilities for Ollama LLM integration
Add your description here
Superduper allows users to work with self-hosted LLM models via [Llama.cpp](https://github.com/ggerganov/llama.cpp).
Chain small language models to outperform large ones — runs locally on 8GB RAM
Swarmauri Llamacpp LLM adapters from swarmauri_standard.
Hardware LLM capability scanner — know what runs on your machine
Python bindings for the tinyllama.cpp inference engine
Modular GGUF patching tool for Ollama multimodal monoliths
MicroPython library for LLM access on MCU
Comprehensive CLI tool and VS Code extension for vision models, AutoML, and local LLMs
Inference locally.
vLLM hardware plugin for Apple Silicon - unifies MLX and PyTorch under a single lowering path
OpenAI and Anthropic compatible LLM inference server for Apple Silicon, powered by MLX
Run local models via vLLM in Docker containers
Ollama-style daemon and CLI over vllm-mlx on Apple Silicon
Tiny local AI agent for edge devices, built to make small models useful with workflows, personas, file/web context, and safe tool-like actions.
A CLI tool to help you decide what kind of LLM you can run locally.
Find the best LLM that runs on your hardware
Python bindings for llama.cpp with native server APIs and prebuilt CPU and GPU wheels
A Python wrapper of llama.cpp
Local native and browser-backed AI inference bridge
helper functions for Ollama with Langchain
Youtube Autonomous Docker Ollama Module
Youtube Autonomous FastAPI Docker Ollama Module
Zero setup, zero config — the easiest Python API for local LLMs on any hardware
LangChain integration for Zeta MLX
Zylon: Your secure, private AI platform, keeping all data in-house.
Local models management utility
Textual TUI for launching and managing llama.cpp servers
Run any AI model on any device, fast, private, and offline.
Run local LLMs on Intel Arc GPUs with one command. Detects your card, installs a verified llama.cpp runtime (Vulkan or SYCL), and serves OpenAI and Ollama compatible APIs.
Offline Expert Agent — local RAG, agent mode, plugins, privacy-first
Run open-source AI on your computer for private chats.
Terminal-native local AI assistant — offline, GPU-accelerated, agentic. Runs any GGUF model locally with full tool use.
Ollama
本地 LLM Runtime 与 GGUF 模型管理工具
Sync Pi coding agent models.json with models served by a vLLM/llama.cpp server
Unofficial local-first diagnostic CLI for NVIDIA NIM developer workflows.
Multi-agent AI assistant with local LLM inference
Production helpers for running Ollama under concurrent load across local and remote endpoints.
Ollama-style GGUF runner with SSD-backed llama.cpp KV cache
A configurable local chatbot library with lightweight memory indexing.
Search and filter Ollama models by size, context, input type and more.
llama.rn provider for Vercel AI SDK
Doraemon Cyber Team — Multi-server Ollama agent
The embeddable woollama model-management core (Rust) — start of the v1.0 port.
Local Ollama text-generation skill for NeuroCore
Terminal-based AI assistant powered by local Ollama with Exa search and Robinhood tools
A desktop GUI application for text classification using LLMs (Ollama, vLLM, SGLang, llama.cpp)
Adds support for LLM locally runned for Capacitor
Find the best LLMs for your hardware specs
Free local AI agent: zero-setup local LLM (llamafile/GGUF), Groq, Gemini, Ollama — HTTP API, sessions, streaming, tool use
LangChain embeddings that run on the Apple Neural Engine, via ANEForge (no CoreML).
LLM plugin for ai-mini-box — local and remote inference, RAG
Polls GoNext cloud API for async local LLM jobs and runs them against Ollama/OpenAI-compatible servers on this Mac
WebLLM provider for Vercel AI SDK v5+ (High-performance in-browser LLM inference)
Run local LLMs on low-RAM machines with automatic hardware detection and streaming inference
GenieX-Bridge Python binding (llama.cpp backend only) — install via pip install geniex-llama-cpp.