Agent frameworks & infrastructure

Local LLM runtimes

Local LLM runtimes run language models on your own hardware instead of calling a hosted API. Ollama handles model downloads and serving, while llama.cpp provides the underlying C/C++ inference engine for GGUF weights. They differ in scope: some are bare runtimes, others bundle chat interfaces or starter setups for private, offline use.

1,401 agents · most starred first

Top 100 of 1,401

ollama★ 187,729

Get up and running with Kimi, GLM, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.

llama.cpp★ 135,821

LLM inference in C/C++

gpt4all★ 77,332

GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.

textgen★ 47,732

Open-source desktop app for local LLMs. Text, vision, tool-calling, OpenAI/Anthropic-compatible API. 100% private.

jan★ 45,934

Jan is an open source alternative to ChatGPT that runs 100% offline on your computer.

BitNet★ 42,307

Official inference framework for 1-bit LLMs

self-hosted-ai-starter-kit★ 15,380

The Self-hosted AI Starter Kit is an open-source template that quickly sets up a local AI environment. Curated by n8n, it provides essential tools for creating secure, self-hosted AI workflows.

cactus★ 14,844

Sub-50mb agentic, voice and vision intelligence for consumer devices.

llama-gpt★ 10,922

A self-hosted, offline, ChatGPT-like chatbot. Powered by Llama 2. 100% private, with no data leaving your device. New: Code Llama support!

petals★ 10,722

🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading

ChatRWKV★ 9,483

ChatRWKV is like ChatGPT but powered by RWKV (100% RNN) language model, and open source.

pinokio★ 7,963

AI Browser

ODS★ 7,857

ODS V3 Pre-Release: Public testing and refinement ahead of the official V3 launch. Turn your PC, Mac, or Linux box into a private AI server.

llama-swap★ 6,196

Reliable model swapping for any local AI server. Chosen by thousands of users. Batteries included. If we don't have it, you probably don't need it.

thunderbolt★ 4,829

AI You Control: Choose your models. Own your data. Eliminate vendor lock-in.

locally-uncensored★ 4,484

The all-in-one local AI studio for your desktop: chat, image and video generation and a coding agent in one free, open source app. Windows and Linux. No Docker, no terminal, no cloud required.

magnitude★ 4,155

Open source inference server that runs the best local models for your hardware, plugged into the agent you already use. Works with Pi, OpenCode, Hermes, OpenClaw, Codex, Claude Code, Oh My Pi, and Cline.

LLamaSharp★ 3,825

A C#/.NET library to run LLM (🦙LLaMA/LLaVA) on your local device efficiently.

atomic-agent★ 3,586

Atomic Agent is a local-first AI agent. Runs open-weight models on your own machine via llama.cpp.

claude-code-local★ 3,523

Hit your Claude Code usage limit? Keep going free and offline on a local model: Qwen 3.8, Gemma 4, and Qwen 4 the day it ships. Runs on Apple Silicon Macs (MLX), Linux and Windows (llama.cpp) with no API key. Your code stays home.

OGAM★ 3,398

The Swiss Army Knife of Offline AI. Chat, see, speak, and generate images on your phone or Mac — GGUF LLMs, vision, Whisper speech-to-text, Stable Diffusion, tool calling, and local-network servers. Runs on your CPU, GPU, or NPU. No account, no API key, zero data leaves your device.

Sidekick★ 3,324

A native macOS app that allows users to chat with a local LLM that can respond with information from files, folders and websites on your Mac without installing any other software. Powered by llama.cpp.

harbor★ 3,278

Stop configuring your AI stack. Start using it. One command brings a complete pre-wired LLM stack with hundreds of services to explore.

maid★ 2,784

Maid is a free and open source application for interfacing with llama.cpp models locally, and with Anthropic, DeepSeek, Ollama, Mistral and OpenAI models remotely.

off-grid-mobile-ai★ 2,503

The Swiss Army Knife of Offline AI. Chat, Speak, and Generate Images - Privacy First, Zero Internet. Download an LLM and use it on your mobile device. No data ever leaves your phone. Supports text-to-text, vision, text-to-image

llama-coder★ 2,082

Replace Copilot local AI

Atomic-Chat★ 2,034

Local AI app and inference engine for agents. Run open-weight LLMs locally — private, 100% offline on your computer. Join our Discord: https://discord.com/invite/8wGSsvmg4V

llama2-webui★ 1,941

Run any Llama 2 locally with gradio UI on GPU or CPU from anywhere (Linux/Windows/Mac). Use `llama2-wrapper` as your local llama2 backend for Generative Agents/Apps.

DreamServer★ 1,792

Turn your PC, Mac, or Linux box into a private AI server. LLM inference, chat UI, voice, agents, workflows, RAG, and image generation.

mlx-serve★ 1,639

Native LLM inference server for Apple Silicon. OpenAI + Anthropic API compatible. No Python. Includes MLX Core macOS app with chat, agent mode, and tool calling.

gemma-chat★ 1,431

Local AI chat + coding agent for Apple Silicon, powered by Gemma 4 via MLX / Supports Ollama

nextjs-ollama-llm-ui★ 1,427

Fully-featured web interface for Ollama LLMs

airunner★ 1,317

Offline inference engine for art, real-time voice conversations, LLM powered chatbots and automated workflows

web-llm-chat★ 1,152

Chat with AI large language models running natively in your browser. Enjoy private, server-free, seamless AI conversations.

z80ai★ 1,140

Z80-μLM is a 2-bit quantized language model small enough to run on an 8-bit Z80 processor. Train conversational models in Python, export them as CP/M .COM binaries, and chat with your vintage computer.

can-it-run-llm★ 1,052

streamlit region:us

mac-code★ 1,035

mac code — Claude Code, but it runs on your Mac for free. 35B AI agent at 30 tok/s via Apple Silicon flash-paging. $0/month.

Box★ 1,030

The most advanced, fully offline client-side AI suite on Android today.

mlxstudio★ 991

MLX Studio - Easiest way to run LLM's on your Mac. All in one engine.

Llama-2-Open-Source-LLM-CPU-Inference★ 972

Running Llama 2 and other Open-Source LLMs on CPU Inference Locally for Document Q&A

MasterAgent★ 965

Build AI agents that run 100% on-device. Sub-100ms latency on Qualcomm NPU. Zero cloud dependency.

selfhost-ai★ 964

One-command installer for a self-hosted AI stack on your own server: n8n, Ollama, Open WebUI, OpenClaw, Dify, Flowise, Supabase, ComfyUI, Qdrant & 30+ tools. Docker Compose, automatic HTTPS via Caddy, built-in monitoring. Free Zapier/Make alternative.

Spark-X2.6★ 924

Spark-x2.6 open model series. Pushing the Limits of Agentic Capabilities in On-Device Models

Qwen3-Coder-Next-GGUF★ 906

gguf qwen3_next unsloth qwen qwen3 text-generation

Llama-2-7B-Chat-GGML★ 873

transformers llama facebook meta pytorch llama-2

deltafin★ 860

Run full Kimi K3 on a single device. And an OpenAI-compatible API server for local chat and coding agents.

llama-node★ 857

Believe in AI democratization. llama for nodejs backed by llama-rs, llama.cpp and rwkv.cpp, work locally on your laptop CPU. support llama/alpaca/gpt4all/vicuna/rwkv model.

llama3.java★ 818

Llama 3+ inference in pure Java

mlx-omni-server★ 746

MLX Omni Server is a local inference server powered by Apple's MLX framework, specifically designed for Apple Silicon (M-series) chips. It implements OpenAI-compatible API endpoints, enabling seamless integration with existing OpenAI SDK clients while leveraging the power of local ML inference.

flutter_edge_ai★ 666

On-device AI for Flutter — Gemma and other LLMs, multimodal, function calling, embeddings, RAG and speech on Android, iOS, Web, macOS, Windows and Linux.

LLM-Hub★ 662

Local LLM, image&video&music generator

pooled★ 658

Peer-to-peer LLM inference in the browser: pool your devices to run big open models, for chat and coding agents.

ChatLLM-Web★ 641

Private local model studio, AI chat, and agent workspace powered by WebGPU and WebLLM.

ChatLLM-Web★ 640

Private local model studio, AI chat, and agent workspace powered by WebGPU and WebLLM.

rkllama★ 630

Ollama alternative for Rockchip NPU: An efficient solution for running AI and Deep learning models on Rockchip devices with optimized NPU support ( rkllm )

ai00_server★ 624

The all-in-one RWKV runtime box with embed, RAG, AI agents, and more.

llmman★ 598

Run any agent on any model, models stored as OCI images

LLaMA_MPS★ 575

Run LLaMA (and Stanford-Alpaca) inference on Apple Silicon GPUs.

hob★ 554

An easier way to use AI on your own computer. A free, open-source alternative to Claude and ChatGPT that runs open-weight models like Gemma and gpt-oss locally on Mac, Windows and Linux. Private, and works offline.

Xolotl★ 545

Xolotl: run DeepSeek-V4.1-Flash, all 552B, as a local coding agent on one gaming PC (RTX 4090 + 110 GB RAM): ~33 tok/s on agent tasks, ~41 in fast mode, 512K context. Plus GLM-5.3-Flash, Qwen3.8 and LFM3-Omni. MoE experts tiered across VRAM, pinned RAM and NVMe, with Xolotl Direct NVMe->VRAM. OpenAI + Anthropic API.

OmniSpace★ 536

本地优先的全栈 AI 创作工作站:AI对话 · AI漫画 · 漫剧创作 · 写作台 · 知识学习五大模块;vLLM / llama.cpp / Transformers 多引擎本地推理,显存智能调度、舱壁隔离自愈、仅供学习使用,欢迎交流指正。(最新进度(除插件系统)受到协议限制,暂时无法提交到仓库)

zinc★ 532

Zig INferenCe Engine — Local LLM inference on AMD GPUs and Apple Silicon

Llama-2-7B-Chat-GGUF★ 517

transformers gguf llama facebook meta pytorch

LLM-Model-VRAM-Calculator★ 517

static region:us

TinyAgent★ 503

[EMNLP 2024 Demo] TinyAgent: Function Calling at the Edge!

catai★ 499

Run AI ✨ assistant locally! with simple API for Node.js 🚀

local-llms-on-android★ 499

Run local LLMs like Gemma, Qwen, and LLaMA on Android for offline, private, real-time chat and question answering with LiteRT and ONNX Runtime.

litert-samples★ 499

LiteRT and LiteRT-LM sample apps, model recipes, agent skills and utilities.

parllama★ 496

TUI for Ollama and other LLM providers

chipper★ 487

✨ AI interface for tinkerers (Ollama, Haystack RAG, Python)

GLM-4.7-Flash-Uncensored-Heretic-NEO-CODE-Imatrix-MAX-GGUF★ 451

gguf GLM 4.7 Flash thinking reasoning NEO Imatrix MAX Quants

Locals★ 451

Run many AI agents at once on your Mac, up to 3x faster.

local-llm-function-calling★ 438

A tool for generating function arguments and choosing what function to call with local LLMs

ROCmFPX★ 435

ROCmFPX Family for AMD Hardware and Processors. More quants and special agent quants

semana-javascript-expert09★ 422

Criando um chatbot inteligente 100% offline e no navegador

Spark-X2.5★ 398

Spark-x2.5 open model series. Pushing the Limits of Agentic Capabilities in On-Device Models

qvac★ 395

QVAC - Local AI SDK and libraries for building private, cross-platform, peer-to-peer AI applications. Run LLMs, speech-to-text, translation, and more locally on Linux, macOS, Windows, Android, and iOS.

NanoLLM★ 380

Optimized local inference for LLMs with HuggingFace-like APIs for quantization, vision/language models, multimodal agents, speech, vector DB, and RAG.

edgen★ 369

⚡ Edgen: Local, private GenAI server alternative to OpenAI. No GPU required. Run AI models locally: LLMs (Llama2, Mistral, Mixtral...), Speech-to-text (whisper) and many others.

TinyLLM★ 357

Setup and run a local LLM and Chatbot using consumer grade hardware.

guaardvark★ 334

Self-hosted AI studio on your own GPU: chat with your files (RAG), screen and browser agents, coding swarms, LoRA training, an MCP server for Claude Code and Cursor, and local video, image, music and voice generation. No telemetry.

PicoMLXServer★ 327

The easiest way to run the fastest MLX-based LLMs locally

DistiLlama★ 305

Chrome Extension to Summarize or Chat with Web Pages/Local Documents Using locally running LLMs. Keep all of your data and conversations private. 🔐

local_llama★ 294

This repo is to showcase how you can run a model locally and offline, free of OpenAI dependencies.

SpeziLLM★ 290

Large Language Model (LLM) module for the Spezi Ecosystem

alzeo★ 270

Alzeo is an open-source installer that turns an NVIDIA DGX Spark, or any GB10 machine (ASUS, Dell, HP, Lenovo, MSI, GIGABYTE, Acer), into a private AI server in one command: measured model lanes, OpenAI and Anthropic APIs, and a local cockpit.

keras-llm-robot★ 251

A web UI Project In order to learn the large language model. This project includes features such as chat, quantization, fine-tuning, prompt engineering templates, and multimodality.

garden-desk★ 249

Private AI on premise

DeepSeek-V4.1-Flash-1017-GSQ-RCO-Coder-GGUF★ 247

gguf text-generation image-text-to-text vision gsq rco

Llamatik★ 224

True on-device AI SDK for Kotlin Multiplatform (Android, iOS, Desktop, JVM, WASM). LLM inference, Speech-to-Text, Image Generation, Agent Framework and RAG — powered by llama.cpp, whisper.cpp and stable-diffusion.cpp.

CASALIOY★ 220

Tiny toolkit for air-gapped LLMs on consumer-grade hardware

Local-LLM-Langchain★ 215

Load local LLMs effortlessly in a Jupyter notebook for testing purposes alongside Langchain or other agents. Contains Oobagooga and KoboldAI versions of the langchain notebooks with examples.

FileWizardAI★ 213

Add AI capabilities to your file system using Ollama, Groq, OpenAi and other's api

Belullama★ 209

Belullama is a comprehensive AI application that bundles Ollama, Open WebUI, and Automatic1111 (Stable Diffusion WebUI) into a single, easy-to-use package.

Llama-2-13B-chat-GGUF★ 206

transformers gguf llama facebook meta pytorch

HuggingFaceGuidedTourForMac★ 205

A guided tour on how to use HuggingFace large language models on Macs with Apple Silicon

Llamatik★ 198

True on-device AI for Kotlin Multiplatform (Android, iOS, Desktop, JVM, WASM). LLM, Speech-to-Text and Image Generation — powered by llama.cpp, whisper.cpp and stable-diffusion.cpp.

deepseek-local-guide-ru★ 197

🐋 Полный гайд на русском: как установить и запустить DeepSeek локально (V4, R1) на ПК, Mac и сервере — Ollama, llama.cpp, LM Studio, vLLM, Open WebUI, API, RAG, агенты, fine-tuning, лучшие фишки

langserve_ollama★ 195

무료로 한국어🇰🇷 파인튜닝 모델 받아서 로컬 LLM 호스팅. LangServe, Ollama, streamlit + RAG

self-hosted-ai-stack★ 180

Deploy a self-hosted AI stack with Docker Compose: local LLMs, chat, document parsing, embeddings, transcription, text-to-speech and MCP tools. Powered by Ollama, LiteLLM, AnythingLLM and more. Local-first with auto-configured authentication, lightweight stacks, optional HTTPS and NVIDIA GPU support. MIT licensed. Multi-arch: amd64, arm64.

More in Agent frameworks & infrastructure