Agent frameworks & infrastructure

Local LLM runtimes

1,401 agents · most starred first

Local LLM runtimes, page 7 of 15

Llama★ 5

🌐 Run GGUF models directly in your web browser using JavaScript and WebAssembly for a seamless and flexible AI experience.

owning-your-agent★ 5

Local agentic coding stack: Hermes Agent + Qwen3.5-27B + GLM-4.7-Flash on dual RTX 3090s. Daily-driver agentic work, no cloud, no metering. Companion to blog.zacharycangemi.com.

mlx-llm-server-mac-m-series★ 5

Local LLM inference server for Qwen models on Apple Silicon. Private, offline-first AI development with OpenCode integration. Or alternatively: Run Qwen models locally on Mac with MLX. Works offline, no subscriptions, full privacy. Integrates with OpenCode for AI-powered development.

symchat★ 5

SymChat is an AI chat assistant designed with privacy as a core principle. All conversations and data are stored locally on your device—nothing is sent to external servers by default. Our goal is to empower users with full control of their AI experience, discourage unnecessary data collection, and ensure your interactions remain private and secure

vscode-ollama-chat★ 5

Experimental extension to chat with models available to download from Ollama

inclave★ 5

Local-first AI assistant for macOS — chat with your PDFs, spreadsheets, CSVs and code using a local LLM via Ollama. Model-generated Python runs in a Seatbelt sandbox with no network. No cloud, no telemetry, no API keys.

claude-code-ollama-sandbox★ 5

Run Claude Code CLI with local Ollama models in an isolated Docker environment. Free, private, offline-capable AI coding assistant.

localmodelrouter★ 5

Local LLM server that provides drop-in API compatibility with both Ollama and OpenAI, using your locally installed [llama.cpp](https://github.com/ggerganov/llama.cpp) 's `llama-server` as the inference backend.

ore-kernel★ 5

The Operating System for Local Intelligence. ⚙️

open-warp★ 5

Run your own LLM inside Warp — local adapter for any OpenAI-compatible provider

leanmeaninferencemachine★ 5

AI agent OS with multiple tools.

Local-LLM-ONNX★ 5

Local LLM desktop application that uses the ONNX Generative AI Runtime. Does not make any networking requests outside of the local machine. Zero HTTP (e.g.: API calls to OpenAI, Gemini), zero REST API middle-layer (e.g.: GPT4All) zero WebSocket middle-layer (Ollama, LM Studio, etc.). Loads a local LLM model.

gpt-oss-tokenizer-playground★ 5

gradio region:us

K2-Chat-GGUF★ 5

gguf text-generation license:apache-2.0 endpoints_compatible region:us imatrix

Vigostral-7B-Chat-GGUF★ 5

transformers gguf mistral LLM finetuned text-generation

Qwen_Qwen3-Coder-480B-A35B-Instruct-GGUF★ 5

gguf text-generation base_model:Qwen/Qwen3-Coder-480B-A35B-Instruct base_model:quantized:Qwen/Qwen3-Coder-480B-A35B-Instruct endpoints_compatible region:us

ollama★ 5

AI model deployment on Synology NAS and macOS 🧠🐳

lumispec.ai★ 5

Local-first AI workbench for llama.cpp models. Full chat UI with SSE streaming, multi-engine web search, sandboxed Python, self-aware source tools with diff previews, hot-loadable plugins, TaskMaster prompt decomposition, two-tier wiki RAG, and inline Chart.js/SVG applets. Runs entirely on your hardware.

Qwen3-Coder-30B-A3B-Instruct-Q4_K_M-GGUF★ 5

transformers gguf llama-cpp gguf-my-repo text-generation base_model:Qwen/Qwen3-Coder-30B-A3B-Instruct

Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF★ 5

gguf llama.cpp rocm gfx1151 strix-halo amd

vault-desk★ 5

Private AI on premise

PocketLM★ 5

Micro-LLM Family: Models From 1KB to 1MB

Yi-1.5-9B-Chat-16K-GGUF★ 5

gguf text-generation base_model:01-ai/Yi-1.5-9B-Chat-16K base_model:quantized:01-ai/Yi-1.5-9B-Chat-16K license:apache-2.0 region:us

small-language-model-chatbot★ 5

The Small Language Model Chatbot is a Python-based interactive chatbot built using Hugging Face's distilGPT-2 model. The chatbot is fun, lightweight, and ideal for demonstrating the capabilities of small-scale language models. It includes interactive widgets for user input and a visual element that tracks and displays .

codegemma-7b-it-GGUF★ 5

transformers gguf text-generation license:gemma endpoints_compatible region:us

Qwen3-Coder-30B-A3B-Instruct-FP8-Q4_K_M-GGUF★ 5

transformers gguf llama-cpp gguf-my-repo text-generation base_model:Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8

codegemma-1.1-7b-it-GGUF★ 5

transformers gguf text-generation base_model:google/codegemma-1.1-7b-it base_model:quantized:google/codegemma-1.1-7b-it license:gemma

garden-desk★ 5

Private AI on premise

utgpt★ 5

An offline AI chat application built specifically for Ubuntu Touch, allowing users to run lightweight Large Language Models (LLMs) directly on their mobile device.

MindDock★ 5

Open-source iOS client for LM Studio, chat with your local LLMs, fully offline and private.

foundry★ 5

foundry project files

M1K3★ 5

Local, offline AI for your Machine. Punk intelligence — no data centers required.

LLaya★ 5

Fast local AI decisions in Lua. A binding for LibLayaX that runs the Laya typed-decision model in-process: no server, no Python. Answers as JSON text or Lua tables. Lua 5.1 to 5.4, on Windows, Linux and macOS. An open-source alternative to Jev

Luwi★ 5

DSH for Android, built on LittleWhale (dsh ported to Android by Miuzarte)

lloyal-ai★ 5

Turn Open Weight models into AI apps

qwen3.8-27b-uncensored-chatbot★ 5

gradio mcp-server region:us

local-llm★ 5

Which model + engine + agent harness runs a coding agent best on local hardware? Real-repo agent benchmarks, pass rates and wall time, measured on an M5 Max MacBook Pro and a two-node DGX Spark cluster. OpenCode is the primary harness.

Lixbon★ 5

LLMs auto-hospedados en un clúster GPU distribuido: gateway compatible con OpenAI, web, desktop, CLI y móvil.

WormGPT-desktop★ 5

Uncensored local AI desktop app for Windows — offline chat, vision, image generation, web search, agent mode & OSINT. No account, no cloud, no telemetry.

UpinelAIOS-MLX★ 5

One-click uncensored AI agent server for Apple Silicon Macs. MLX + MTPLX speculative decoding, OpenAI-compatible on your LAN. Up to 79.4 t/s decode, MTP speculative decoding on.

phonon★ 5

Old phones as inference servers for local agentic AI.

apple-silicon-ai-notes★ 4
pygbnf★ 4

A composable Python DSL for building GBNF grammars compatible with llama.cpp.

quick-llama★ 4

Run Ollama models on Google Colab

Caduceus★ 4

Infinite power but in a pendrive

ollama-local-serve★ 4

Local LLM infrastructure for distributed AI applications. Serve Ollama-powered models across your network

easy-langchain★ 4

langchain use ollama api and qwen2.5

GIA★ 4

Run local or API connected models from OpenAI, Hugging Face, or OpenRouter. Use your own data, load .safetensors or GPTQ models, and extend capabilities with Python plugins. Full UI (Gradio) and CLI integration. No cloud lock in. No coding required.

chatllm.rs★ 4

rust api wrapper for llm-inference chatllm.cpp

ChatMeld-Ollama★ 4

Privacy-focused multi-LLM chat app with Ollama support for local AI models (Qwen, Llama, Mistral). Fork of ChatMeld.

cheap_meets_mistralAI★ 4

When cheap meets Mistral AI

llava-docker★ 4

Docker image for LLaVA: Large Language and Vision Assistant

aeon.ai★ 4

AEON is portable, private, and capable of operating fully offline (with the exception of web search). It democratizes access to powerful, dynamic AI capabilities for a wider audience, regardless of their hardware.

chatllm.nim★ 4

Nim api-wrapper for llm-inference chatllm.cpp

spring-aI-with-local-LLAMA3★ 4

This demo demonstrates the integration of a local LLama3 model for handling JSON responses within the Spring AI framework, positioning the model as a 'business logic' executor

openvino-agent-stack★ 4

Run Qwen3-8B at ~18 tok/s on a Core Ultra laptop iGPU. Local-only LLM + VLM + ReAct agent stack with $0 token cost. Drop-in Claude Code backend.

nextcloud-local-llm★ 4

Local LLM chat application for Nextcloud with on-premise AI, full data sovereignty and enterprise-grade security.

chat-app★ 4

Local Chat App using Ollama, FastHTML, Redis & OAuth

LLM-local★ 4

Technical documentation and implementation roadmaps for a local LLM, with clear separation of responsibilities between Infrastructure and Development teams.

filamind-ai★ 4

Private AI for your Synology NAS — local LLM chat with optional cloud fallback. Multi-user, full Arabic + English UI, OpenAI-compatible API. Supports DVA 3221 (CUDA GPU) and DS1821+ (CPU AVX2).

portable-ai★ 4

Private, portable AI that runs from a USB drive. Mac + Windows. No internet. No installation. No subscription.

llmigo★ 4

One-shot Windows and Linux installers for LLM-ready development tooling

LLM_deployment_zerocost_playground★ 4

gradio region:us

llm-jp-4-8b-thinking★ 4

gradio region:us

gpt2-home★ 4

streamlit region:us

anon8231489123-gpt4-x-alpaca-13b-native-4bit-128g★ 4

gradio region:us

GPT-2★ 4

gradio region:us

On-Device-Function-Calling★ 4

gradio region:us

Qwen3-Coder-480B-A35B-Instruct-MLX-4bit★ 4

transformers safetensors qwen3_moe text-generation mlx conversational

Qwen3-Coder-480B-A35B-Instruct-MLX-8bit★ 4

transformers safetensors qwen3_moe text-generation mlx conversational

Qwopus3.6-27B-Coder-imatrix-MTP-GGUF★ 4

gguf llama.cpp text-generation text-generation-inference transformers quantization

DeepSeek-Coder-V2-Instruct-0724-GGUF★ 4

gguf text-generation base_model:deepseek-ai/DeepSeek-Coder-V2-Instruct-0724 base_model:quantized:deepseek-ai/DeepSeek-Coder-V2-Instruct-0724 license:other endpo

Qwen2.5-Coder-0.5B-Instruct-GGUF★ 4

gguf code codeqwen chat qwen qwen-coder

MiniMax-M3-REAP22-Coder★ 4

mlx safetensors minimax_m3_vl vmlx jang reap

MiniMax-M3-Coder-Small★ 4

mlx safetensors minimax_m3_vl jang reap awq

speechless-tora-code-7B-v1.0-GGUF★ 4

transformers gguf llama llama-2 code text-generation

Qwable-9B-Claude-Fable-5-OBLITERATED-GGUF★ 4

docker region:us

TinyLlama-1.1B-Chat-v0.4-GGUF★ 4

gguf ggml quantized q2_k q3_k_m q4_k_m

pastura★ 4

AIgazing. Like stargazing, but for local LLMs.

BAMChat★ 4

Basic and Modifiable Chat, based on Ollama

av-local-llm-api★ 4

Allows to easily run local REST API with a custom LLM, running locally or remotely, with user defined system instructions. Useful for quick local autmations that require problem solving with large langague models and interaction via a REST API.

vinos★ 4

An agentic Linux distro — local LLMs + Claude Code one keystroke away. Arch + Hyprland, boots on any x86_64 (T2 Intel Macs included). Ship an AI-native desktop in 15 min from a USB.

openbeast★ 4

Your own private AI workstation — frontier-class local models, a 15-tool agent, and secure remote access, all self-hosted on your GPU. No cloud, no API keys.

aps-ollma-chatbot★ 4

Try Run APS use local LLM Llama3

OpenZero-Gemma4-E2B-Agentic-GGUF★ 4

llama.cpp gguf gemma local-ai agentic-ai coding-assistant

internlm2_5-7b-chat-1m-gguf★ 4

gguf chat text-generation en license:apache-2.0 endpoints_compatible

Yi-1.5-6B-Chat-GGUF★ 4

gguf llama text-generation base_model:01-ai/Yi-1.5-6B-Chat base_model:quantized:01-ai/Yi-1.5-6B-Chat license:apache-2.0

lim★ 4

Stateful local LLM inference manager with persistent KV-cache, native filesystem tools, and instant undo: built in C++ on llama.cpp

Qwen2.5-Coder-3B-Instruct-Q8_0-GGUF★ 4

transformers gguf code codeqwen chat qwen

streamlit_local_llm★ 4

Example streamlit UI to talk to a local llm (for instance run via lm studio)

CodeQwen1.5-7B-GGUF★ 4

gguf pretrained text-generation en license:other endpoints_compatible

CodeLlama-7b-Instruct-hf-GGUF★ 4

transformers gguf code llama llama-2 text-generation-inference

entropic★ 4

Local-first agentic inference engine in C/C++. Multi-tier model routing, grammar-constrained output, MCP tool servers. Embeddable via C ABI.

xiaoxiai-android★ 4

离线 Android 端侧 AI:语音识别 / 翻译 / TTS / 端侧大模型全本地推理,数据不出设备 · On-device Android AI: offline ASR, translation, TTS & LLM via ONNX Runtime, zero cloud

pocket-agent★ 4

Run tool-calling LLM agents locally on Android devices using Termux and llama.cpp with complete data privacy and no external API reliance.

VLS-32M-Context-Local-LLM★ 4

Give Claude Code, Codex and opencode a 32M-token memory that never leaves your machine. Local LLM server (MCP + OpenAI API) that runs any GGUF model with a context far larger than its window - no cloud, no retraining, no vector database. Windows and Linux, one command.

Sailor2-1B-Chat-GGUF★ 4

gguf mistral quantized 2-bit 3-bit 4-bit

kvmem-agent-16gb★ 4

Reproducible 16GB-VRAM Windows local-agent stack: DSH Desktop + KVMem (llama.cpp fork, KV cache in system RAM) + parameter panel. Flagship: Bonsai 2 CRACK 27B — 100+ tok/s, 262K ctx on RTX 4080. GSQ for intelligence at ~50 tok/s; QQZ default.

LLauncher★ 4

Lightweight hardware tuner, profile manager, and local model inference launcher for llama.cpp (`llama-server.exe').

sovereign-cloud-devsummit26★ 4

Leak Hunt: hands-on demo of bringing AI to your data. A .NET assistant, an egress traffic monitor, Foundry Local and a browser lab. Synthetic data only.

More in Agent frameworks & infrastructure