Agent frameworks & infrastructure

Local LLM runtimes

1,401 agents · most starred first

Local LLM runtimes, page 13 of 15

petallm

PetaLLM allows single 4GB GPU card to run 70B large language models without quantization, distillation or pruning. 8GB vmem to run 405B Llama3.1.

picollm

picoLLM Inference Engine

picollmdemo

picoLLM Inference Engine demos

pip-boy-llm

PIP-BOY LLM - Vault-Tec Local AI Terminal

pixelarraycloudstorage

PixelArray 像素云盘客户端

polyollama

Manage multiple Ollama servers and fan out parallel inference across them.

pullama-cli

Fix ollama pull TLS timeout and disconnects — resumable Ollama model downloader for slow connections

pyagentkit

Agent toolkit for Ollama

pydantic-ai-lmstudio-provider

Add your description here

pydantic-ai-mlx

Run MLX compatible HuggingFace models with Pydantic AI locally

pydanticai-ollama

Pydantic AI Ollama integration

pygpt4all

Official Python CPU inference for GPT4All language models based on llama.cpp and ggml

pygptj

Python bindings for the GGML GPT-J Laguage model

pyllama2

pyllama2

pyllamacpp

Python bindings for llama.cpp

pymllm

Fast and lightweight multimodal LLM inference engine for mobile and edge devices

pythonaibrain-llm

LLM extension for Pythonaibrain with quantized model support

quantgpt-local-runner

Local runner for QuantGPT with WebSocket support

quick-llama

Run Ollama models easily, anywhere – including online platforms like Google Colab

quick-ollama

fast startup ollama server

qv-ollama-sdk

A simple SDK for interacting with the Ollama API by automatically creating a conversation (chat history)

rabbitllm

Run 70B+ LLMs on a single 4GB GPU — no quantization required. Layer-streaming inference for consumer hardware.

real-llama-cpp-python

A simple custom LLM wrapper for llama.cpp with LangChain compatibility.

reworkd-platform-Ollama

A platform for reworkd

rkLLM
rocky-llm-python-amki

LLM in python

runllama

Simple CLI runner for Ollama (LLM) models

scikit-ollama

Scikit-Ollama: an extension of Scikit-LLM for Ollama served models.

self-llm

A package for using self-served LLM

serapeum-llama-cpp

llama cpp integration for serapeum

serapeum-ollama

Ollama integration for Serapeum

sillm-mlx

Running and training LLMs on Apple Silicon via MLX

simple-vllm

VLLM & Unsloth Wrapper for simple local inference.

sinapsis-llama-cpp

Sinapsis templates for llama.cpp text generation.

sllm-lib

Super Lazy Language Model

slullama

Shared Ollama gateway over Slurm — wake a GPU node on demand

smartifyLLM

A package that enhances local LLMs with real-time web awareness and adaptive context retrieval, making them smarter with no retraining or fine-tuning.

smollama-py

A python app to serve smOllama

st-ollama

A Streamlit chatbot app integrating Ollama LLMs

stache-ai-ollama

Ollama provider for Stache AI

strands-microgpt

Karpathy's pure-Python microGPT as a Strands Agents model provider — zero dependencies, pure autograd, from scratch.

streamlit-ollama

Streamlit utilities for Ollama LLM integration

sucrose-ollama

Add your description here

superduper-llamacpp

Superduper allows users to work with self-hosted LLM models via [Llama.cpp](https://github.com/ggerganov/llama.cpp).

sutra-llm

Chain small language models to outperform large ones — runs locally on 8GB RAM

swarmauri_llm_llamacpp

Swarmauri Llamacpp LLM adapters from swarmauri_standard.

syko-llm
tinillm

Hardware LLM capability scanner — know what runs on your machine

tinyllama-cpp

Python bindings for the tinyllama.cpp inference engine

tsunagi-ollama-bridge

Modular GGUF patching tool for Ollama multimodal monoliths

uLLMpy

MicroPython library for LLM access on MCU

vllama

Comprehensive CLI tool and VS Code extension for vision models, AutoML, and local LLMs

vllm-inference

Inference locally.

vllm-metal

vLLM hardware plugin for Apple Silicon - unifies MLX and PyTorch under a single lowering path

vllm-mlx

OpenAI and Anthropic compatible LLM inference server for Apple Silicon, powered by MLX

vllmd

Run local models via vLLM in Docker containers

vllmlx

Ollama-style daemon and CLI over vllm-mlx on Apple Silicon

weirenong-simpleagent

Tiny local AI agent for edge devices, built to make small models useful with workflows, personas, file/web context, and safe tool-like actions.

whatllm

A CLI tool to help you decide what kind of LLM you can run locally.

whichllm

Find the best LLM that runs on your hardware

xllamacpp

Python bindings for llama.cpp with native server APIs and prebuilt CPU and GPU wheels

xllamacpp-cuda12x

A Python wrapper of llama.cpp

xlocllm

Local native and browser-backed AI inference bridge

yollama

helper functions for Ollama with Langchain

yta-docker-ollama

Youtube Autonomous Docker Ollama Module

yta-fastapi-docker-ollama

Youtube Autonomous FastAPI Docker Ollama Module

zerollm-kit

Zero setup, zero config — the easiest Python API for local LLMs on any hardware

zeta-mlx-langchain

LangChain integration for Zeta MLX

Zylon AI

Zylon: Your secure, private AI platform, keeping all data in-house.

lazyllama

Local models management utility

llama-tui

Textual TUI for launching and managing llama.cpp servers

Nexa AI

Run any AI model on any device, fast, private, and offline.

arc-llama

Run local LLMs on Intel Arc GPUs with one command. Detects your card, installs a verified llama.cpp runtime (Vulkan or SYCL), and serves OpenAI and Ollama compatible APIs.

ollama-term

Offline Expert Agent — local RAG, agent mode, plugins, privacy-first

Jan AI

Run open-source AI on your computer for private chats.

hyperlite-ai

Terminal-native local AI assistant — offline, GPU-accelerated, agentic. Runs any GGUF model locally with full tool use.

@agentic-kit/ollama

Ollama

llm-nest-python

本地 LLM Runtime 与 GGUF 模型管理工具

pi-codingagent-sync

Sync Pi coding agent models.json with models served by a vLLM/llama.cpp server

nim-doctor

Unofficial local-first diagnostic CLI for NVIDIA NIM developer workflows.

solenoid-node

Multi-agent AI assistant with local LLM inference

ollama-orchestra

Production helpers for running Ollama under concurrent load across local and remote endpoints.

diskllm

Ollama-style GGUF runner with SSD-backed llama.cpp KV cache

llama-simple-chat-bot

A configurable local chatbot library with lightweight memory indexing.

ollama-model-search

Search and filter Ollama models by size, context, input type and more.

@react-native-ai/llama

llama.rn provider for Vercel AI SDK

dct-agent

Doraemon Cyber Team — Multi-server Ollama agent

woollama-core

The embeddable woollama model-management core (Rust) — start of the v1.0 port.

neurocore-skill-ollama

Local Ollama text-generation skill for NeuroCore

ollama-robin

Terminal-based AI assistant powered by local Ollama with Exa search and Robinhood tools

ollama-classifier-gui

A desktop GUI application for text classification using LLMs (Ollama, vLLM, SGLang, llama.cpp)

@capgo/capacitor-llm

Adds support for LLM locally runned for Capacitor

spec2llm

Find the best LLMs for your hardware specs

freeaiagent

Free local AI agent: zero-setup local LLM (llamafile/GGUF), Groq, Gemini, Ollama — HTTP API, sessions, streaming, tool use

langchain-aneforge

LangChain embeddings that run on the Apple Neural Engine, via ANEForge (no CoreML).

ai-mini-box-llm

LLM plugin for ai-mini-box — local and remote inference, RAG

@tiens.nguyen/gonext-local-worker

Polls GoNext cloud API for async local LLM jobs and runs them against Ollama/OpenAI-compatible servers on this Mac

@browser-ai/web-llm

WebLLM provider for Vercel AI SDK v5+ (High-performance in-browser LLM inference)

digvijay-llm

Run local LLMs on low-RAM machines with automatic hardware detection and streaming inference

geniex-llama-cpp

GenieX-Bridge Python binding (llama.cpp backend only) — install via pip install geniex-llama-cpp.

More in Agent frameworks & infrastructure