Agent frameworks & infrastructure
Local LLM runtimes
1,401 agents · most starred first
Local LLM runtimes, page 7 of 15
🌐 Run GGUF models directly in your web browser using JavaScript and WebAssembly for a seamless and flexible AI experience.
Local agentic coding stack: Hermes Agent + Qwen3.5-27B + GLM-4.7-Flash on dual RTX 3090s. Daily-driver agentic work, no cloud, no metering. Companion to blog.zacharycangemi.com.
Local LLM inference server for Qwen models on Apple Silicon. Private, offline-first AI development with OpenCode integration. Or alternatively: Run Qwen models locally on Mac with MLX. Works offline, no subscriptions, full privacy. Integrates with OpenCode for AI-powered development.
SymChat is an AI chat assistant designed with privacy as a core principle. All conversations and data are stored locally on your device—nothing is sent to external servers by default. Our goal is to empower users with full control of their AI experience, discourage unnecessary data collection, and ensure your interactions remain private and secure
Experimental extension to chat with models available to download from Ollama
Local-first AI assistant for macOS — chat with your PDFs, spreadsheets, CSVs and code using a local LLM via Ollama. Model-generated Python runs in a Seatbelt sandbox with no network. No cloud, no telemetry, no API keys.
Run Claude Code CLI with local Ollama models in an isolated Docker environment. Free, private, offline-capable AI coding assistant.
Local LLM server that provides drop-in API compatibility with both Ollama and OpenAI, using your locally installed [llama.cpp](https://github.com/ggerganov/llama.cpp) 's `llama-server` as the inference backend.
The Operating System for Local Intelligence. ⚙️
Run your own LLM inside Warp — local adapter for any OpenAI-compatible provider
AI agent OS with multiple tools.
Local LLM desktop application that uses the ONNX Generative AI Runtime. Does not make any networking requests outside of the local machine. Zero HTTP (e.g.: API calls to OpenAI, Gemini), zero REST API middle-layer (e.g.: GPT4All) zero WebSocket middle-layer (Ollama, LM Studio, etc.). Loads a local LLM model.
gradio region:us
gguf text-generation license:apache-2.0 endpoints_compatible region:us imatrix
transformers gguf mistral LLM finetuned text-generation
gguf text-generation base_model:Qwen/Qwen3-Coder-480B-A35B-Instruct base_model:quantized:Qwen/Qwen3-Coder-480B-A35B-Instruct endpoints_compatible region:us
AI model deployment on Synology NAS and macOS 🧠🐳
Local-first AI workbench for llama.cpp models. Full chat UI with SSE streaming, multi-engine web search, sandboxed Python, self-aware source tools with diff previews, hot-loadable plugins, TaskMaster prompt decomposition, two-tier wiki RAG, and inline Chart.js/SVG applets. Runs entirely on your hardware.
transformers gguf llama-cpp gguf-my-repo text-generation base_model:Qwen/Qwen3-Coder-30B-A3B-Instruct
gguf llama.cpp rocm gfx1151 strix-halo amd
Private AI on premise
Micro-LLM Family: Models From 1KB to 1MB
gguf text-generation base_model:01-ai/Yi-1.5-9B-Chat-16K base_model:quantized:01-ai/Yi-1.5-9B-Chat-16K license:apache-2.0 region:us
The Small Language Model Chatbot is a Python-based interactive chatbot built using Hugging Face's distilGPT-2 model. The chatbot is fun, lightweight, and ideal for demonstrating the capabilities of small-scale language models. It includes interactive widgets for user input and a visual element that tracks and displays .
transformers gguf text-generation license:gemma endpoints_compatible region:us
transformers gguf llama-cpp gguf-my-repo text-generation base_model:Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8
transformers gguf text-generation base_model:google/codegemma-1.1-7b-it base_model:quantized:google/codegemma-1.1-7b-it license:gemma
Private AI on premise
An offline AI chat application built specifically for Ubuntu Touch, allowing users to run lightweight Large Language Models (LLMs) directly on their mobile device.
Open-source iOS client for LM Studio, chat with your local LLMs, fully offline and private.
foundry project files
Local, offline AI for your Machine. Punk intelligence — no data centers required.
Fast local AI decisions in Lua. A binding for LibLayaX that runs the Laya typed-decision model in-process: no server, no Python. Answers as JSON text or Lua tables. Lua 5.1 to 5.4, on Windows, Linux and macOS. An open-source alternative to Jev
DSH for Android, built on LittleWhale (dsh ported to Android by Miuzarte)
Turn Open Weight models into AI apps
gradio mcp-server region:us
Which model + engine + agent harness runs a coding agent best on local hardware? Real-repo agent benchmarks, pass rates and wall time, measured on an M5 Max MacBook Pro and a two-node DGX Spark cluster. OpenCode is the primary harness.
LLMs auto-hospedados en un clúster GPU distribuido: gateway compatible con OpenAI, web, desktop, CLI y móvil.
Uncensored local AI desktop app for Windows — offline chat, vision, image generation, web search, agent mode & OSINT. No account, no cloud, no telemetry.
One-click uncensored AI agent server for Apple Silicon Macs. MLX + MTPLX speculative decoding, OpenAI-compatible on your LAN. Up to 79.4 t/s decode, MTP speculative decoding on.
Old phones as inference servers for local agentic AI.
A composable Python DSL for building GBNF grammars compatible with llama.cpp.
Run Ollama models on Google Colab
Infinite power but in a pendrive
Local LLM infrastructure for distributed AI applications. Serve Ollama-powered models across your network
langchain use ollama api and qwen2.5
Run local or API connected models from OpenAI, Hugging Face, or OpenRouter. Use your own data, load .safetensors or GPTQ models, and extend capabilities with Python plugins. Full UI (Gradio) and CLI integration. No cloud lock in. No coding required.
rust api wrapper for llm-inference chatllm.cpp
Privacy-focused multi-LLM chat app with Ollama support for local AI models (Qwen, Llama, Mistral). Fork of ChatMeld.
When cheap meets Mistral AI
Docker image for LLaVA: Large Language and Vision Assistant
AEON is portable, private, and capable of operating fully offline (with the exception of web search). It democratizes access to powerful, dynamic AI capabilities for a wider audience, regardless of their hardware.
Nim api-wrapper for llm-inference chatllm.cpp
This demo demonstrates the integration of a local LLama3 model for handling JSON responses within the Spring AI framework, positioning the model as a 'business logic' executor
Run Qwen3-8B at ~18 tok/s on a Core Ultra laptop iGPU. Local-only LLM + VLM + ReAct agent stack with $0 token cost. Drop-in Claude Code backend.
Local LLM chat application for Nextcloud with on-premise AI, full data sovereignty and enterprise-grade security.
Local Chat App using Ollama, FastHTML, Redis & OAuth
Technical documentation and implementation roadmaps for a local LLM, with clear separation of responsibilities between Infrastructure and Development teams.
Private AI for your Synology NAS — local LLM chat with optional cloud fallback. Multi-user, full Arabic + English UI, OpenAI-compatible API. Supports DVA 3221 (CUDA GPU) and DS1821+ (CPU AVX2).
Private, portable AI that runs from a USB drive. Mac + Windows. No internet. No installation. No subscription.
One-shot Windows and Linux installers for LLM-ready development tooling
gradio region:us
gradio region:us
streamlit region:us
gradio region:us
gradio region:us
gradio region:us
transformers safetensors qwen3_moe text-generation mlx conversational
transformers safetensors qwen3_moe text-generation mlx conversational
gguf llama.cpp text-generation text-generation-inference transformers quantization
gguf text-generation base_model:deepseek-ai/DeepSeek-Coder-V2-Instruct-0724 base_model:quantized:deepseek-ai/DeepSeek-Coder-V2-Instruct-0724 license:other endpo
gguf code codeqwen chat qwen qwen-coder
mlx safetensors minimax_m3_vl vmlx jang reap
mlx safetensors minimax_m3_vl jang reap awq
transformers gguf llama llama-2 code text-generation
docker region:us
gguf ggml quantized q2_k q3_k_m q4_k_m
AIgazing. Like stargazing, but for local LLMs.
Basic and Modifiable Chat, based on Ollama
Allows to easily run local REST API with a custom LLM, running locally or remotely, with user defined system instructions. Useful for quick local autmations that require problem solving with large langague models and interaction via a REST API.
An agentic Linux distro — local LLMs + Claude Code one keystroke away. Arch + Hyprland, boots on any x86_64 (T2 Intel Macs included). Ship an AI-native desktop in 15 min from a USB.
Your own private AI workstation — frontier-class local models, a 15-tool agent, and secure remote access, all self-hosted on your GPU. No cloud, no API keys.
Try Run APS use local LLM Llama3
llama.cpp gguf gemma local-ai agentic-ai coding-assistant
gguf chat text-generation en license:apache-2.0 endpoints_compatible
gguf llama text-generation base_model:01-ai/Yi-1.5-6B-Chat base_model:quantized:01-ai/Yi-1.5-6B-Chat license:apache-2.0
Stateful local LLM inference manager with persistent KV-cache, native filesystem tools, and instant undo: built in C++ on llama.cpp
transformers gguf code codeqwen chat qwen
Example streamlit UI to talk to a local llm (for instance run via lm studio)
gguf pretrained text-generation en license:other endpoints_compatible
transformers gguf code llama llama-2 text-generation-inference
Local-first agentic inference engine in C/C++. Multi-tier model routing, grammar-constrained output, MCP tool servers. Embeddable via C ABI.
离线 Android 端侧 AI:语音识别 / 翻译 / TTS / 端侧大模型全本地推理,数据不出设备 · On-device Android AI: offline ASR, translation, TTS & LLM via ONNX Runtime, zero cloud
Run tool-calling LLM agents locally on Android devices using Termux and llama.cpp with complete data privacy and no external API reliance.
Give Claude Code, Codex and opencode a 32M-token memory that never leaves your machine. Local LLM server (MCP + OpenAI API) that runs any GGUF model with a context far larger than its window - no cloud, no retraining, no vector database. Windows and Linux, one command.
gguf mistral quantized 2-bit 3-bit 4-bit
Reproducible 16GB-VRAM Windows local-agent stack: DSH Desktop + KVMem (llama.cpp fork, KV cache in system RAM) + parameter panel. Flagship: Bonsai 2 CRACK 27B — 100+ tok/s, 262K ctx on RTX 4080. GSQ for intelligence at ~50 tok/s; QQZ default.
Lightweight hardware tuner, profile manager, and local model inference launcher for llama.cpp (`llama-server.exe').
Leak Hunt: hands-on demo of bringing AI to your data. A .NET assistant, an egress traffic monitor, Foundry Local and a browser lab. Synthetic data only.