Agent frameworks & infrastructure

Local LLM runtimes

1,401 agents · most starred first

Local LLM runtimes, page 2 of 15

PocketStrike★ 173

PocketStrike is a privacy-first mobile AI workstation. It combines zero-cloud native on-device GGUF inference (ARM NEON accelerated) │ with frontier cloud LLMs, multi-step ReAct agent tools, Model Context Protocol (MCP) servers, self-evolving cross-session memory, and FLUX neural image │ generation.

booster★ 170

Booster - open accelerator for LLM models. Better inference and debugging for AI hackers

Qwen3-Coder-30B-A3B-Instruct-1M-GGUF★ 170

transformers gguf unsloth qwen3 qwen text-generation

Llama3-8B-Chinese-Chat-GGUF-8bit★ 166

transformers gguf llama text-generation llama-factory orpo

Sharp-Spark-X2.5-4B-GGUF★ 165

gguf llama.cpp imatrix spark2_5 long-context coding

Agents-A1-Q4_K_M-GGUF★ 155

gguf quantized moe vlm vision agentic

LumaBrowser★ 154

Local AI desktop app: a Chromium browser with a GPU-aware llama.cpp runtime, OpenAI-compatible API, AI chat and agents, Stable Diffusion image/video, TTS and music, plus an MCP server so Claude Code, Codex and Cline can drive your browser. Windows, macOS, Linux.

CodeLlama-7B-Instruct-GGUF★ 153

transformers gguf llama llama-2 text-generation code

gpt4local★ 152

Openai-style, fast & lightweight local language model inference w/ documents

HilbertRaum★ 150

AI, fully local on your computer.

open-genai★ 142

デジタル庁のガバメントAI「源内(GENAI)」を完全ローカル(ローカルLLM/OpenAI互換)で動かす非公式プロジェクト。SAML認証(Keycloak)・RAG(Qdrant)・文字起こし(Whisper)・画像生成(SD)・チーム単位ナレッジをローカル完結。

Llama-2-7b-chat-coreml★ 136

transformers coreml llama text-generation meta llama-2

CodeLlama-7B-GGUF★ 135

transformers gguf llama llama-2 text-generation code

CodeLlama-13B-Instruct-GGUF★ 133

transformers gguf llama llama-2 text-generation code

Qwen2.5-Coder-32B-Instruct-GGUF★ 132

gguf code codeqwen chat qwen qwen-coder

Beledarians_LM_Studio_Toolbox★ 130

This project is a plugin for LM Studio that provides a rich set of tools to a large language model. It allows the LLM to interact with your local file system, execute code, access the internet, and more. This empowers the LLM to perform complex tasks like code generation, project scaffolding, and web research.

Qwen2.5.1-Coder-7B-Instruct-GGUF★ 130

gguf code codeqwen chat qwen qwen-coder

jetson-copilot★ 129

A reference application for a local AI assistant with LLM and RAG

ollama4j-web-ui★ 128

Web UI for Ollama built in Java with Vaadin, Spring Boot and Ollama4j

MLX-Outil★ 125

Tool calling using MLX Swift across iOS, macOS, and visionOS platforms

local-ai-code-assistant★ 125

Local AI Coding Hub: Run Claude Models Offline in 2026

LocalAgent★ 121

Local Reasoning Agent using LangChain + Ollama

dots★ 121

Hyprdots, Hyprland Omarchy, Quickshell, Agentic Local-Ai, Custom Integrated, Minimal

Llama-2-70B-Chat-GGUF★ 121

transformers gguf llama facebook meta pytorch

AirClaw★ 120

Run OpenClaw on a local model. Zero API cost. One OpenAI-compatible endpoint over Ollama, llama.cpp, vLLM or LM Studio — plus AirLLM for models bigger than your GPU.

Oxide-Lab★ 119

Modern desktop application (Rust + Tauri v2 + Svelte 5 + Candle (HF)) for communicating with AI models that runs completely locally on your computer. No subscriptions, no data sent to the internet — just you and your personal AI assistant

servllama★ 117

One app for both cloud LLMs and on-device LLMs.

local-chat★ 116

LocalChat is a ChatGPT-like chat that runs on your computer

CodeQwen1.5-7B-Chat-GGUF★ 116

gguf chat text-generation en arxiv:2309.16609 license:other

VocalClaw_11Labs★ 115

Run OpenClaw AI agent with zero API cost, local LLM via AirLLM

InferrLM★ 115

InferrLM - On-device AI for iOS & Android

ajean★ 115

A personal assistant that remembers you, in one single executable: connect the API of your choice (OpenAI, Anthropic, Mistral, Ollama…). Memory, tasks, apps, browser control and tools it builds itself. Windows, macOS, Linux.

Qwen2.5-Coder-7B-Instruct-abliterated-GGUF★ 112

gguf chat abliterated uncensored text-generation en

TwoAI★ 111

A simple experiment on letting two local LLM have a conversation about anything!

pasllm★ 109

PasLLM - LLM inference engine in Object Pascal (synced from my private work repository)

local-ai-mac★ 109

Practical guide to running LLMs locally on Apple Silicon, with a focus on architecture, memory scaling, and agent workflows.

CodeLlama-34B-Instruct-GGUF★ 108

transformers gguf llama llama-2 text-generation code

ovo-local-llm★ 105

A private Claude-Code-style coding agent for Apple Silicon — run chat, code, and local model workflows on-device. MLX-native, Ollama/OpenAI API compatible, zero API keys.

llmaker★ 105

Selfhost modern LLM stacks. Run the whole fleet from your terminal

mimimodel★ 102

MimiModel: Agentic LLM on a $5 chip. 100% on-device. Private by design.

MiniCPM5-1B-Agentic-Tooluse-GGUF★ 100

gguf llama.cpp llama-cpp minicpm minicpm5 minicpm5-1b

LLMinator★ 99

Gradio based tool to run opensource LLM models directly from Huggingface

Eris.★ 99

Eris is a private AI chat application that runs entirely on your device using Apple's MLX framework. Named after the dwarf planet that challenged our understanding of the solar system, Eris challenges the notion that AI must live in the cloud.

Local-Hermes-Portable★ 99

Premium, zero-configuration local AI inference & agentic workspace running completely offline on Windows, macOS, and Linux.

local_AI_code_assistant_tutorial★ 98

Guide on how to set up LMStudio and VSCode Continue.dev extension as a local and private coding Copilot. Targeted for 32GB RAM Apple Silicon devices.

opilot★ 96

🦙 Opilot: Ollama for GitHub Copilot VS Code Extension – Run Ollama models locally and in the cloud within GitHub Copilot Chat

Agent-RAG-LLM_deploy★ 95
opilot★ 94

🦙 Opilot: Ollama for GitHub Copilot VS Code Extension – Run Ollama models locally and in the cloud within GitHub Copilot Chat

InferrLM★ 92

InferrLM - On-device AI for iOS & Android

qwen3.8-27b-ninfer-5070tilaptop-kvmem-local-agent★ 92

RTX 5070 Ti Laptop 12GB 跑 Qwen3.8-27B:NInfer + KVMem、256K 逻辑上下文、RAM KV、MTP、快速缓存续接与 dsh 本地 Agent / 搜索 / 浏览器。含源码、Windows 部署及实测。

Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF★ 90

gguf rocmfp4 rocmfpx vulkan strix-halo qwen4exp

homl★ 89

The easiest & fastest way to run LLMs in your home lab

Qwen3.8-Flash-Next-Gyro-Coder-GGUF★ 89

gguf qwen3.8-flash-next qwen4exp code experimental agentionai

edge-veda★ 87

On-device AI SDK for Flutter — LLM inference, vision, STT, TTS, image generation, embeddings, RAG, and function calling. Metal GPU on iOS/macOS.

PyOMlx★ 85

A wannabe Ollama equivalent for Apple MlX models

Llama-2-7b-chat-mlx★ 85

mlx llama facebook meta llama-2 text-generation

accuretta★ 85

A sovereign, zero-trust local AI agent and micro-IDE.

gemma-jetson-demos★ 83

Gemma models running 100% local on an 8 GB Jetson Orin Nano: voice assistants with RAG, multi-agent NPCs, live EN-JA translation, and a Gemma 4 voice + vision agent that decides when to look through the webcam. The demos from the Gemma 2 launch at Google Tokyo.

GPT2-ChineseChat-NCNN★ 82

GPT2⚡NCNN⚡中文对话⚡x86⚡Android

CleverWick★ 82

Electron + Next.js desktop AI assistant that runs GGUF models locally using llama.cpp. Designed for offline use, portability, and zero-install deployment.

Your-First-LLM-Studio★ 81

First LLM Studio: local-first LLM workbench for Apple Silicon + MLX. 本地优先 LLM 工作台,覆盖 Agent、对比、微调、评测与运行时遥测。

server★ 80

Self-hosted agentic AI platform powered by local models

omarchy-local-ai★ 80

Local AI for Omarchy: the model validated for your GPU, one button on the bar. Start serves it, any coding agent opens on it, one click shares it on your tailnet.

mafdet★ 80

You type. It already knows. Local LLMs that read and think while you type.

imp★ 79

Run large AI chat models privately on your own NVIDIA RTX 50 or RTX PRO graphics card. On one RTX 5090, Qwen3.8-27B writes about 70 words per second, 1.4x faster than vLLM. Even a model almost four times larger than the card's memory runs. Works with apps made for ChatGPT or Claude. Starts with one docker command.

internlm2_5-7b-chat-gguf★ 79

gguf chat text-generation en license:apache-2.0 endpoints_compatible

TandemLLM★ 79

An environment where open models work as agents: a first-level harness with routing, skills, memory and sandboxed tools, on an engine that drafts and verifies its own tokens.

local-agents★ 78

Local Agents is an add-on for Godot 4.5 to run LLMs locally in games

greenboost-windows★ 78

Windows port of GreenBoost - GPU VRAM extension with system RAM & NVMe for larger LLMs. Original by Ferran Duarri (GPL v2).

Qwen1.5-7B-Chat-GGUF★ 77

gguf chat text-generation en arxiv:2309.16609 license:other

MiniCPM5-1B-Agentic-Tooluse-GGUF★ 77

gguf llama.cpp llama-cpp minicpm minicpm5 minicpm5-1b

vicuna.ps1★ 76

This is a PowerShell script that automates the process of setting up and running VICUNA on a CPU (without a graphics card) using the llama.cpp library and a pre-trained ggml-vicuna-13b-4bit.bin model. The script downloads and extracts the required files, creates a batch file to run VICUNA, and creates a desktop shortcut to launch the batch file.

gguf-loader★ 76

GGUF Loader with its Agentic Mode, and floating button, ai Models | Open Source & Offline. Mistral, Deepseek, llama, gemma, qwen

GPT-OSS-WebGPU★ 76

static region:us

trashclaw★ 75

Local LLM agent for vintage Mac hardware. Built on a 2013 Mac Pro trashcan with dual AMD FirePro D500s. Elyan Labs.

localharness★ 75

An open-source, coding agent harness designed specifically for local LLMs. Designed as everything a user needs to jumpstart their own harness (detailed event steams, logs, and performance data) or quickly replace closed source one (featuring out of the box plugins for web search, image gen, memory, auto research and more in dev).

Yi-34B-Chat-GGUF★ 75

transformers gguf yi text-generation base_model:01-ai/Yi-34B-Chat base_model:quantized:01-ai/Yi-34B-Chat

browser-llm★ 74

Browser LLM Demo (like ChatGPT). Works locally with JavaScript and WebGPU

LlamaLib★ 74

Cross-Platform High-Level LLM Library

ollama-client★ 74

Ollama Client – Chat with Local LLMs Inside Your Browser A lightweight, privacy‑first extension to chat with local LLMs via Ollama, LM Studio, and llama.cpp. Supports streaming, stop/regenerate, RAG, and easy model switching — all without cloud APIs or data leaks.

PrivatePixelPrompt★ 74

PortableLM — Zero-dependency, air-gapped local AI that runs entirely from a USB drive. Chat, generate images, and synthesize speech offline on Windows, macOS, or Linux. No installs, no internet, just plug and play.

llmedge★ 73

Android native AI inference library, bringing text, image, video, STT, TTS inference

zero-cost-cline★ 73

Tired of paying for frontier AI models? Break free with this guide to 100% local, private LLM code generation on Apple Silicon. Optimized for 24GB+ Macs, it uses Ollama + Cline to build a disciplined, free agentic workflow. Learn to write system rules, configure custom Modelfiles, and run strict two-layer test suites.

Qwen1.5-14B-Chat-GGUF★ 72

gguf chat text-generation en arxiv:2309.16609 license:other

Jan-code-4b-gguf★ 72

transformers gguf agent text-generation en base_model:janhq/Jan-code-4b

OpenClaw-Ollama★ 71

OpenClaw with Ollama is a lightweight, high-performance integration designed to run sophisticated AI agents locally on Android devices. By leveraging the power of Termux and the flexibility of Ollama, this project enables a completely private, free, and offline-capable AI workstation in your pocket.

lilbee★ 71

A local model engine with a search engine and web crawler built in. Run your coding agents on your own GPUs, and get cited answers from your files, code, and the sites you crawl. One executable.

LocalMode★ 71

Local-first AI toolkit for the modern web. Build privacy-first AI applications that work entirely in the browser.

LFM2-1.2B-RAG-GGUF★ 70

transformers gguf liquid lfm2 edge text-generation

Sharp-MiniCPM5-2B-GGUF★ 69

gguf llama.cpp imatrix minicpm long-context coding

llm.f90★ 68

LLM inference in Fortran

Jack-3.8-27B-Coder-16GB-VRAM★ 68

llama.cpp gguf 27b jack jack-xml agentic

Qwen1.5-72B-Chat-GGUF★ 68

gguf chat text-generation en arxiv:2309.16609 license:other

pyllamacpp★ 67

Python bindings for llama.cpp

ollama-mcp-agent★ 66

Ollama MCP Agent allows you to use LLM models locally on your PC for free along with MCP additional features

OfflineAI-KMP★ 66
web-ai-toolkit★ 65

The Web AI Toolkit is a powerful, privacy-first JavaScript library that brings advanced AI capabilities directly to your web applications. Run OCR, speech-to-text, text summarization, image classification, and more — all locally in the browser with no data sent to external servers.

offgrid-tools★ 65

Self-contained offline environment providing local AI chat, offline Wikipedia/content archives, IRC communication, audio streaming, file server, and development tools. Designed for zero internet dependency - download once, run anywhere. Perfect for remote areas, emergency scenarios, or escaping surveillance capitalism.

node-llmatic★ 64

Use self-hosted LLMs with an OpenAI compatible API

Project-Shard★ 63

Make AI Free Again - Shard is a GUI for Ollama LLM's. Powered by Next.js.

More in Agent frameworks & infrastructure