Agent frameworks & infrastructure

Local LLM runtimes

1,401 agents · most starred first

Local LLM runtimes, page 3 of 15

mlxchat★ 61

A ChatGPT-like app using Alibaba's Qwen3.5 models locally on the phone - with tool calling & vision

pico-jarvis★ 60
copaw-docker★ 60

QwenPaw(原 CoPaw 项目)的 Docker 部署方案,支持一键构建和运行,相比官方镜像更小。

pondok-ai★ 60

Turn one machine into a private ChatGPT for a whole room. One command sets up Ollama, Open WebUI, RAG and an OpenAI-compatible API sized to your hardware, and pondok-ai share lets a classroom or office chat from their own phones.

FerryAI★ 59

Native AI inference for PHP 8.3+ - run ONNX, GGUF (llama.cpp) and RubixML models directly in your PHP process via FFI. Chat, streaming, embeddings, RAG and vector search. No Python, no HTTP microservices, no Docker sidecars.

using-llama3-locally★ 58

Running llama3 using Ollama-Python, Curl, LangChain, Chroma, and User interface.

node-configs★ 58

Find node configurations to easily setup your Gaia nodes on your machine with open source models and ready-made settings.

Qwen1.5-32B-Chat-GGUF★ 58

gguf chat text-generation en arxiv:2309.16609 license:other

Chaty★ 58

The models on your disk, put to work. A private desktop app for open models — chat, a coding agent built for small models, text-to-image, text-to-music, answers from your own documents, and voice. GGUF on llama.cpp, MLX on Apple Silicon. macOS, Windows, Linux. No account, no cloud, no telemetry.

yalla★ 57

A tiny LLM Agent with minimal dependencies, focused on local inference.

turbofit★ 57

Local-first adaptive LLM runtime for Hermes Agent. Matches 8–300 GB hardware, acquires pinned GGUFs through Turbohaul, and safely contracts and heals under VRAM pressure—no remote fallback by default.

ClusterChat★ 57

A chatbot, with a large language model split across its users, entirely in the browser.

CodeLlama-34B-GGUF★ 56

transformers gguf llama llama-2 text-generation code

samosa-chat★ 55

Run large models like Qwen3.6-35B-A3B locally on a 16 GB RAM machine

auto-ollama★ 54

run ollama & gguf easily with a single command

csglite★ 53

CSGLite is a lightweight local LLM inference platform. One command downloads, loads, and chats with models. It ships a web UI, OpenAI-compatible API, llama.cpp inference, resumable downloads, marketplace browsing, and one-click AI app and coding-agent setup—all in a single cross-platform binary.

localcloud★ 51

Stop paying for AI APIs during development. LocalCloud runs everything locally - GPT-level models, databases, all free.

Qwen3-Coder-30B-A3B-Instruct-GGUF★ 51

gguf text-generation base_model:Qwen/Qwen3-Coder-30B-A3B-Instruct base_model:quantized:Qwen/Qwen3-Coder-30B-A3B-Instruct endpoints_compatible region:us

omp-ninfer★ 51

Durable local inference for Oh My Pi: NInfer + Qwen3.8 27B on one RTX 3090/4090/5090, with restart-resumable OpenAI Responses state.

Ollama-Workbench★ 50

A comprehensive platform for managing, testing, and leveraging Ollama AI models with advanced features for customization, workflow automation, and collaborative development.

AGmind★ 50

Legacy: self-hosted LLM/RAG stack installer for DGX Spark (arm64). The work continued as AGmind Systems Lab — measured local-LLM performance evidence (Strix Halo, DGX Spark, llama.cpp, vLLM): https://agmind.ai

tt-studio★ 50

The Tenstorrent Studio (TT-Studio) is an easy to use web interface for running AI models on Tenstorrent hardware. It handles all the technical setup automatically and gives you a simple GUI to deploy models, chat with models, and more.

adm★ 50

ADM (Auto Deploy Model) 基于 Tauri 2.x 构建,将 llama.cpp 复杂的命令行启动指令通过简洁的图形界面呈现,让你在本地轻松部署、运行大语言模型,并内置 Agent 终端 同时支持把本地模型和云端模型接入智能体工作流。

monkeys-with-typewriters★ 49

The complete AI platform on a $3 microcontroller. Sub-millisecond inference. Zero hallucinations.

larri★ 49

L.A.R.R.I. = Local Agent for Remote Rigging of Inference.

python-ai-agent-ollama★ 49

A local AI agent built from scratch in Python with Ollama: conversation memory, system prompts, and multi-step tool calling.

All-Agents★ 49

Install and run every AI agent on your phone locally

cerebras_Qwen3-Coder-REAP-25B-A3B-GGUF★ 48

gguf text-generation base_model:cerebras/Qwen3-Coder-REAP-25B-A3B base_model:quantized:cerebras/Qwen3-Coder-REAP-25B-A3B endpoints_compatible region:us

Aion-Instruct-Preview-Sample★ 48

Sample app showing on-device language model inference with the Aion Instruct Preview Framework on Windows Copilot+ PCs.

open-llm-webui★ 47

This repository contains a web application designed to execute relatively compact, locally-operated Large Language Models (LLMs).

use-ollama-to-enhance-claude★ 47

Pair Claude Desktop on Anthropic with Claude Code routed through Ollama. Visual walkthrough + copy-paste prompt that cuts your Claude Code bill ~90%.

Qwen1.5-0.5B-Chat-GGUF★ 47

gguf chat text-generation en arxiv:2309.16609 license:other

wingman★ 46

Wingman is the fastest and easiest way to run Llama models on your PC or Mac.

gemma_4_tool_calling★ 46

A straightforward Python guide demonstrating how to build a local, agentic AI assistant using Gemma 4, Ollama, and real-time API function calling.

Qwen3-Coder-480B-A35B-Instruct-1M-GGUF★ 46

transformers gguf unsloth qwen3 qwen text-generation

coreai-kit★ 46

Run LLMs, VLMs, ASR, TTS, diarization and more fully on-device with Apple's Core AI framework (iOS/macOS 27) — one line of Swift per model, 60 models pinned to immutable Hugging Face revisions, each gated against its source model before enrollment. Plugs into Apple's FoundationModels API: LanguageModelSession with tool calling + guided generation.

naidan★ 45

A privacy-focused, local-first AI app. Run models on your device, directly in your browser, or connect to OpenAI-compatible APIs and Ollama. It also runs via file://.

CodeLlama-70B-Python-GGUF★ 45

transformers gguf llama llama-2 text-generation code

lmstudio-copilot-provider★ 44

Use local LM Studio models in VS Code Copilot Chat with streaming, tool calling, terminal/file tools, and optional image generation.

Qwen3-Coder-30B-A3B-Instruct-MLX-4bit★ 44

transformers safetensors qwen3_moe text-generation mlx conversational

openmake_llm★ 44

Open-source, self-hosted AI workspace for local and open-weight LLMs with vLLM, LiteLLM, autonomous agents, MCP tools, deep research, artifacts, and BYOK.

ai-hardware-fit★ 44

Can my GPU run this? Pick a GPU to get 3 fitting AI models, quantization, VRAM, speed ranges, and run commands. 한국어·English.

lychee★ 43

Universal local LLM runtime. Drop-in Ollama replacement with HuggingFace integration, pipelines, structured output. Free & open source.

llm-gpu-checker-ko★ 43

Find local AI models for your GPU. Open-source VRAM, quantization and speed estimates in your browser — no install or sign-up. 한국어·English.

kaggle-llm-server★ 43

Run a free 35B multimodal LLM on Kaggle GPU — accessible from any OpenAI-compatible client

Pithos★ 43

A dependency-free C inference engine for Liquid AI's LFM2 / LFM2.5 models. Terminal chat, an OpenAI-compatible server, a web agent and a terminal agent, all in one executable.

Qwen3-Coder-30B-A3B-Instruct-GGUF★ 42

transformers gguf qwen qwen3 qwen3-coder byteshape

CodeLlama-70B-hf-GGUF★ 42

transformers gguf llama llama-2 text-generation code

modulo★ 42

Modulo lets agents running local models try modifying their own activations to change their inner state. Changes decay after a certain time and can be undone at any time.

docker-ai-stack★ 41

Deploy a complete, self-hosted AI stack on your own server with one command. Includes Ollama (LLM), AnythingLLM (chat UI), LiteLLM (AI gateway), Whisper (STT), Kokoro (TTS), Embeddings (RAG), and MCP Gateway. Most services run locally; LiteLLM optionally routes to external providers. Multi-arch: amd64, arm64. NVIDIA GPU (CUDA) acceleration.

local-ai-work-agent-template★ 41
KMP-MineGPT★ 41

MineGPT 是一个基于Kotlin Multiplatform 开发的本地小型语言模型(SLM)对话应用; MineGPT is a lightweight local SLM (Small Language Model) chat application built with Kotlin Multiplatform. It aims to provide a cross-platform and user-friendly AI assistant experience.

Llama-3-ChatQA-1.5-8B-GGUF★ 41

gguf nvidia chatqa-1.5 chatqa llama-3 pytorch

agentica-org_DeepScaleR-1.5B-Preview-GGUF★ 41

gguf text-generation en dataset:AI-MO/NuminaMath-CoT dataset:KbsdJames/Omni-MATH dataset:RUC-AIBOX/STILL-3-Preview-RL-Data

Qwen2.5-Coder-3B-Instruct-abliterated-GGUF★ 41

gguf code codeqwen chat qwen qwen-coder

dsh-harmonyos-pc★ 41

让 DeepSeek Harness(dsh) 在 HarmonyOS 完整跑起来:缓存优化对话预设 + 启动补丁 + 工具链 + 鸿蒙桌面/平板端客户端,node 不锁版本(最新线 Node 26.10.0、LTS 24.21.0) | Run DeepSeek Harness (dsh) on HarmonyOS: presets, patches, tooling & native desktop/tablet clients; node not pinned (latest line 26.10.0 / LTS 24.21.0)

MAGI★ 40

AI system powered by large language models.

Qwen3.8-Flash-Next-AP-GGUF★ 40

gguf qwen3.8-flash-next qwen4exp imatrix agentionai text-generation

commonplace★ 40

Offline AI research assistant for Android. Answers from Wikipedia and other reference packs on the phone, with citations. Gemma 4 runs on the Pixel 10's Tensor NPU (first word in about 2 s); other phones use llama.cpp. No INTERNET permission.

crew-llamafile★ 39

Run CrewAI agent workflows on local LLM models with Llamafile and Ollama

continuum★ 39

Local-first AI citizens on your own hardware. Teams of continuously-learning personas (Rust core, llama.cpp fork, LoRA genomes) that collaborate, remember, dream, and improve on public benchmarks — on a MacBook. Not assistants you rent: minds you host.

internlm2_5-20b-chat-gguf★ 39

gguf chat text-generation en license:apache-2.0 endpoints_compatible

gptq_for_langchain★ 38

A guide about how to use GPTQ models with langchain

DeepSeek-V4.1-Flash-1017-Uncensored-GSQ-RCO-Coder-GGUF★ 38

gguf deepseek-v4.1 gsq-rco coder abliterated uncensored

Cortex★ 37

Local first mini-harness fully capable of simple every-day coding or admin tasks using Ollama/gguf

local-llm-autotune★ 37

Zero-config local LLM optimization for Ollama, LM Studio, and Apple Silicon MLX. Reduces TTFT by 40%, wall time for local agents by 46%, and RAM usage by 3x.

gemma-4-E4B-Agentic-Opus-Reasoning-GeminiCLI-mlx-4bit★ 37

mlx gemma4 safetensors 4-bit quantized apple-silicon

strands-mlx★ 36

Experimental: MLX model provider for Strands Agents - Build, train, and deploy AI agents on Apple Silicon.

Lynn★ 36

Lynn GitHub 镜像仓 · Qwen3.8-27B Coder390 EfficientThink 本地 GGUF(llama.cpp Q2/Q3,内置 Q4 MTP)· 主仓 https://github.com/MerkyorLynn/Lynn · 下载 https://download.merkyorlynn.com/download.html

gemma-4-26B-A4B-it-assistant-GGUF★ 36

gguf atomic-chat gemma gemma4 google llama.cpp

DeepSeek-Coder-V2-Instruct-GGUF★ 36

gguf text-generation base_model:deepseek-ai/DeepSeek-Coder-V2-Instruct base_model:quantized:deepseek-ai/DeepSeek-Coder-V2-Instruct license:other endpoints_compa

cyllama★ 35

A thin cython wrapper around llama.cpp, whisper.cpp and stable-diffusion.cpp

quaynor★ 35

AI inference library for mobile devices

ILLIP★ 35

Your AI company, in your device. Local-first AI platform: chat, agent crew, memory, and a model store sized to your hardware - from 4GB laptops to big GPUs. One-click setup, fully private.

libai★ 34

A C library for embedding Apple Intelligence on-device Foundation models in any programming language or application with full support for native tool calling and MCP.

pi-sbx-llamacpp★ 34

Run Pi coding agent isolated in a Docker Sandbox microVM with a local llama-server as the inference backend

gemma-4-E4B-Agentic-Opus-Reasoning-GeminiCLI-GGUF★ 34

gguf gemma4 quantized multimodal vision reasoning

Llama-3-Groq-8B-Tool-Use-GGUF★ 34

gguf facebook meta pytorch llama llama-3

Qwen2.5-Coder-7B-Instruct-4bit★ 34

transformers safetensors qwen2 text-generation code codeqwen

ash4d-local-ai-agent-hub★ 34

Self-hosted AI architecture for 2026 running on a k3s Kubernetes cluster, providing local Qwen3 model inference, vector RAG pipelines, AI agent frameworks, and hardware-accelerated image generation driven by GitOps workflows.

Lunar_Astral_Agents★ 34

纯本地化桌面 AI 智能体平台。Go + TypeScript + C/C++ 全栈,集成多模态对话、文生图、语音识别与合成,零 Python / 零云端依赖,全部推理本地完成。

Mingbird-agent★ 34

Local-first agent harness (Windows + Ollama) for 2–9B models. Same 2B model: 0.017 → 0.821 across four agent harnesses — 48×. 288-cell benchmark, every cell public. One-click zero-outbound mode. 本地优先,一键断网,零遥测。鸣鸟

velocity-local-services★ 34

Velocity Local Services: an app for running LLMs locally on Windows and Linux. Load any GGUF model and chat with web search, tools and schematics, let it use your computer by voice, and serve it to opencode, Hermes or any app through an OpenAI-compatible API and MCP. Context up to 32M tokens. No cloud.

local_agentic_llm★ 33

local_agentic_llm

Ai-Systems-New★ 33

On-device AI SDK powering ToolNeuron — LLM chat & tool calling (llama.cpp), Stable Diffusion image generation (QNN/MNN), image processing (upscale, segment, inpaint, depth, style), and TTS. Native C++ + Kotlin JNI. Fork or clone to use in your own app.

Invarail★ 33

The authority plane for local AI agents: a governed, self-hosted agent built so a 7–30B model on your own GPU can drive it. Measured in public.

DGX-Model-Manager★ 32

Single-file web UI for NVIDIA DGX Spark — pull Ollama models, browse and download from HuggingFace, manage LiteLLM routing, and control SGLang, vLLM, llama.cpp, LocalAI, and ComfyUI. All from one browser tab.

hermes-claude-code-local★ 32

Run Hermes Agent + Claude Code locally on llama.cpp — zero API costs. A 4h / 7M-token session that would have cost $94 on Claude Opus 4.7

qwen3.8-flash-coder-26gb-gguf★ 32

gguf llama-cpp ollama moe qwen code

ollama-agent★ 31

Ollama Agent is a powerful command-line tool (CLI and REPL) that allows you to interact with local AI models.

NIKA_desktop★ 31

Native AI assistant for macOS, built with SwiftUI and llama.cpp. Run local LLMs offline, work with PDFs, and create custom AI assistants. Optional NIKA cloud integration.

gemma3-ollama-tools★ 30

This project shows how to unlock Ollama tool-calling with Google’s Gemma 3, which doesn't natively support this feature. Includes practical examples, custom prompts, and model-file adjustments to demonstrate effective integration.

p8hub★ 29

Private AI Hub (P8Hub) - Host and use your own AI Services. Keep everything simple and private.

ai-suite★ 29

AI-Suite - n8n, OpenClaw, Open WebUI, OpenCode, Llama.cpp/Ollama, Flowise, Langfuse, MCP Gateway and more!

Qwen1.5-1.8B-Chat-GGUF★ 29

gguf chat text-generation en arxiv:2309.16609 license:other

Qwen3-Coder-Next-8bit★ 29

mlx safetensors qwen3_next text-generation conversational base_model:Qwen/Qwen3-Coder-Next

ishizuki★ 29

works well with rocks; native Apple Silicon inference engine for capable and tiny models (ternary, Qwen) via MLX Swift. Hadamard-rotated KV cache, vision, tool calling, OpenAI + Anthropic APIs

ollama-docker-web-application★ 28

Xây dựng AI Agent Website

ollama-functions★ 28

Ollama function calling demo

AGmind64★ 28

Self-hosted LLM/RAG stack in one command — AMD Strix Halo / x86_64 (ROCm/Vulkan, Docker Compose)

More in Agent frameworks & infrastructure