Agent frameworks & infrastructure
Local LLM runtimes
1,401 agents · most starred first
Local LLM runtimes, page 5 of 15
A self-hosted personal chatbot API with FastAPI & Langchain, It allows you to interact with the Llama2 LLM (and other open-source LLMs) to have natural language conversations, generate text, and perform various language-related tasks.
This repo contains a code that uses colabxterm and langchain community packages to install Ollama on Google Colab free tier T4 and pulls a model from Ollama and chats with it
Langchain integration package with LocalAI. It provides embeddings and reranker support
Local AI for Apple Silicon. Chat with open LLMs (Qwen, Gemma, DeepSeek, Mistral, Llama…) fully offline on your Mac via Apple MLX — private by design, zero telemetry.
mlx safetensors gemma4_assistant speculative-decoding mtp gemma
mlx safetensors qwen3_5_moe qwen world-model agent
static region:us
Fun project to run your own LLM chat bot using llama.cpp
A bridge between JetBrains Koog framework and on-device LLM inference engines
gguf quantized GGUF imatrix quantization imat
transformers gguf text-generation zh license:apache-2.0 region:us
transformers gguf llama text-generation code text-generation-inference
gguf text-generation pt en es ru
docker region:us
Rusty inference library using various backends including llama.cpp
A tiered-memory system design for workloads that don't fit in RAM: measure the working set, pin the hot tier, stream the cold tier from flash. Ships the residency calculator, measurement harnesses, and the build recipes behind it. Predictions validated against public benchmarks.
⚡ Foq — the FREE, local, open-source alternative to Jev. Typed System 1 decisions in ~25 ms — no waitlist, no cloud, no per-token cost. foq.fr
ClippyAI is a multi-platform GUI application, which serves as an interface for Ollama.
NVIDIA DGX Spark ressources
docker region:us
gguf qwen-agentworld abliterated 4-bit quantized llama.cpp
**A fully local, privacy-first LLM desktop application powered by llama.cpp**
gguf jack-xml j-space agentic long-context qwen3.6
gguf ternary quantization integer llama.cpp taardis
InferCrate: self-hosted local LLM server powered by Ollama. Secure by default with auto-generated Bearer token auth. Native Ollama API and an OpenAI-compatible /v1 API subset. First-start model pre-pull, NVIDIA GPU (CUDA) acceleration, and persistent model storage. Docker deployment on CPU (amd64/arm64) or CUDA (amd64).
Fullstack chatbot built using Rust. Made using Candle, Leptos, Actix, Tokio and Tailwind. Uses quantized Mistral 7B Instruct v0.1 GGUF models.
Local LLM Chatbot GUI. Interacts with open-source models offline via Ollama. Built in Python with a sleek interface. / Interfaz grafica local para chatear offline con modelos open-source usando Ollama.
A step-by-step guide to setting up and running DeepSeek locally using Ollama, making it accessible via a browser as an offline chatbot.
Chat bot based on Ollama
SharpAI is an embeddable embeddings, completions, and model management platform using llama.cpp via LlamaSharp, with a built-in webserver with Ollama and OpenAI APIs.
Chat with GGUF LLMs using llama.cpp and a classic Windows Forms interface for minimal GUI bloat.
Docker image to run an Ollama local LLM server. Secure by default, all API requests require a Bearer token (auto-generated on first start). OpenAI-compatible API. Supports first-start model pre-pull, NVIDIA GPU (CUDA) acceleration, and persistent model storage. Multi-arch: amd64, arm64.
High-performance Android SDK for on-device LLM inference (GGUF). Privacy-focused, offline-first, and powered by llama.cpp with a clean Kotlin Coroutines API.
Programmable Responsive Interface for Smart Models
Little gguf - a tiny, embeddable, fully functional direct quantized inference engine for llama/qwen models in standard gguf format; no dependencies
Fully autonomous offline NLP AI (ELIZA) for M5Stack Cardputer ADV.
Curated resource for mobile teams shipping on-device LLMs — runtime benchmarks, model picks, GDPR-friendly architecture, and real production use cases.
Skill для Hermes / Claude Code / Codex-агентов: автоматический выбор и настройка LLM inference engine под железо, модель и workload — MLX, llama.cpp, ExLlama, vLLM, SGLang, TensorRT-LLM.
🇺🇦 AI-HomeLab: Домашні AI-Лабораторії в Україні — локальний ШІ, мультиагентні системи, блекаут-резилієнтність
gradio region:us
transformers safetensors llama text-generation en dataset:cerebras/SlimPajama-627B
gguf text-generation base_model:01-ai/Yi-1.5-9B-Chat base_model:quantized:01-ai/Yi-1.5-9B-Chat license:apache-2.0 endpoints_compatible
gguf ggml quantized q2_k q3_k_m q4_k_m
Run xAI's Grok CLI from any project, with all traffic routed through a VPN (Docker + gluetun, Mullvad by default)
Kotlin Multiplatform engine for running Gemma LLMs on-device on Android via LiteRT-LM — stateful KV-cache chat sessions, resumable model management, function calling. Includes NativeLM, a private Local AI chat app. AGPL-3.0 / commercial.
"Local LLM agent framework in Rust — native tool-role support, tmux session persistence, two-tier summarization, and ShareGPT logging. Works with any OpenAI-compatible backend."
An Android-native local AI runtime with MCP tools, phone capabilities, and OpenAI-compatible APIs.
让 DeepSeek Harness(dsh) 在 HarmonyOS 完整跑起来:缓存优化对话预设 + 启动补丁 + 工具链 | Run DeepSeek Harness on HarmonyOS: cache-optimized agent presets, patches & tooling
Single-binary GGUF model runtime in C — CPU/CUDA/Metal, OpenAI-compatible. Serves, scores, and trains LoRA directly through the quantized weights it deploys, with byte-reproducible adapters. Tool calls survive the token limit; sparse MoE and schema-constrained decoding included.
transformers gguf text-generation license:gemma endpoints_compatible region:us
A blazing-fast, local-first AI agent ecosystem. Orchestrate local LLMs with Python, run them in a native Rust terminal, or integrate directly into your editor via ACP.
Fast local AI decisions inside your application. A shared library and C API built on laya.cpp that runs the Laya typed-decision model in-process: no server, no Python. Windows, Linux and macOS, x64 and ARM64, CPU or GPU. An open-source alternative to Jev
Adaptive Cognitive Budgeting (ACB) dynamically allocates context windows and CPU threads based on query complexity and host memory pressure to prevent catastrophic latency stalls in GPU-free LLM inference.
An open-source, on-device AI assistant for Android. Chat with GPT, Claude, Gemini or your own local models, keep long-term memory on your phone, and let the AI use real tools. No account, no tracking, no telemetry. Built with Kotlin and Material 3 Expressive.
Flint: a local-first agentic workspace, a fork of Jan
transformers gguf llama llama-2 code text-generation
Evidence-first frugal AI: small language models and pipelines on an old laptop or Raspberry Pi, no cloud, no GPU. Measured benchmarks and recipes, with Indian-language focus.
Kotlin Multiplatform application showcase with llama.cpp on device LLM inference & Koog.ai agent
Run local LLMs powered up by tools in Telegram Messenger
Examples using the llmedge library
Sovereign Local AI Workspace & LLM Runtime Manager. Easily Run GGUF, MLX, OpenAI, Gemini, etc. Under One Unified Interface ✨
A tiny, headless, AI-native operating system: a minimal Linux + musl/BusyBox appliance that boots straight into an on-device LLM agent over a local HTTP API and serial console. Local-first inference with a Rust agent core, Go gateway, and per-service cgroup/seccomp/Landlock isolation.
mlx safetensors qwen2 code codeqwen chat
gguf nvidia chatqa-1.5 chatqa llama-3 pytorch
mlx safetensors qwen3_5 qwen3_6 token-efficient efficient-thinking
gguf GLM 4.7 Flash thinking reasoning NEO Imatrix MAX Quants
MoziAI-35B: Local open-source financial AI LLM. 35B MoE model compressed to 15.5GB via MoziSmartBit quantization. 256K context, vision, tool calling, 140+ tok/s on consumer GPU. Based on Qwen3.6-35B-A3B.
A lightweight, memory-budgeted GGUF inference runtime and developer playground for running small language models on constrained devices.
The Seed: the smallest version of Node0, rung by rung, built on a bench with an agent: pages, costs, pitfalls, measurements and the configuration as it ran. Mirror of the Forgejo origin.
EZ AI — Local AI made easy: Windows command center for local AI apps, model library, and coding agents.
PHP extension for local inference from pre-trained models.
DSH for Android, built on LittleWhale (dsh ported to Android by Miuzarte)
gguf lfm3 claude stepbystep agent toolcall
在 Android 本地运行 DeepSeek Harness,支持插件、悬浮对话与 Root / Shizuku。Run DeepSeek Harness locally on Android.
Peridot v1.6.0 [STABLE] | Sovereign AI Kernel. Offline, GPU accelerated Qwen framework: a 27B model fully on an 8GB GPU at 23.6 tok/s. Sandboxed skills & plugins, model tool use, opt in web search, MCP bridge for private data jobs from cloud agents, ZAT-SCS preemption, RAG. Zero data exfiltration. Engineered by uncoalesced.
Create a chatbot in less than 2 minutes with MLX Server, Chainlit and Llama3
Terminal-based AI chatbot with infinite personalities, using local LLMs via ollama
An Ollama client built with Svelte 5 and SvelteKit. Chat completely local and client-side with a friendly interface!
GUI Running DeepSeek's reasoning model distilled from DeepSeek-R1 based on Llama and Qwen.
Node-RED Flows for OpenAI API compatible endpoints calling llama.cpp
A collection of intelligent AI agents built using Ollama, LangChain, and local LLMs, including chatbots, voice assistants, web scrapers, document readers, and specialized tools. No API required. Fully autonomous and privacy-preserving.
A production-grade Python SDK for the Lemonade LLM backend. Featuring auto-discovery, port scanning, and a low-overhead client architecture. Powering the Sorana AI workspace.
Local-first GenAI lab with a React frontend and Spring Boot backend, supporting Ollama, Amazon Bedrock, Hugging Face, RAG document retrieval, MCP-backed AWS tools, streaming chat, persistent sessions, and artifact/report workflows.
NVIDIA AI Hub — One-click AI app launcher for NVIDIA DGX GPU. Install, run, and manage GPU-accelerated AI apps from a modern web UI.
Nemotron-3-Super-120B Cogni-Brain on DGX Spark: 131K context, ~24 tok/s, 93/100 Tool-Eval, hardened agent stack.
Run large Mixture-of-Experts LLMs that exceed system RAM on Apple Silicon by loading only router-selected experts from SSD with MLX. Includes OpenAI/Anthropic-compatible serving for local agentic coding.
Run PaperClip On HuggingFace Space
Self-hosted Open source AI platform with RAG, TTS, fully offline
gguf llama 1b text-generation local-llm llama-cpp
mlx safetensors qwen2 code codeqwen chat
gradio region:us
transformers gguf gpt llm large language model h2o-llmstudio
mlx safetensors kimi_k3 moe reap pruned
Speed dating for local AI models: find the best one your PC can actually run. Tests Ollama models on your own hardware.
transformers gguf gpt llm large language model h2o-llmstudio
transformers gguf text-generation license:gemma endpoints_compatible region:us
Field-tested local AI stack optimizations for HermesAgent, LM Studio, and Qwen 3 models
gguf code qwen qwen-coder codeqwen text-generation
Lightweight standalone LLM inference engine. Single binary, no runtime deps. Serves GGUF models via Ollama/OpenAI/MCP APIs. Multi-agent orchestration, TUI, 21 built-in tools.
transformers gguf code codeqwen chat qwen