Agent frameworks & infrastructure

Local LLM runtimes

1,401 agents · most starred first

Local LLM runtimes, page 5 of 15

llm-practice★ 13

A self-hosted personal chatbot API with FastAPI & Langchain, It allows you to interact with the Llama2 LLM (and other open-source LLMs) to have natural language conversations, generate text, and perform various language-related tasks.

Ollama_GoogleColab_colabxterm_langchain★ 13

This repo contains a code that uses colabxterm and langchain community packages to install Ollama on Google Colab free tier T4 and pulls a model from Ollama and chats with it

langchain-localai★ 13

Langchain integration package with LocalAI. It provides embeddings and reranker support

Ka1zen★ 13

Local AI for Apple Silicon. Chat with open LLMs (Qwen, Gemma, DeepSeek, Mistral, Llama…) fully offline on your Mac via Apple MLX — private by design, zero telemetry.

gemma-4-E4B-it-assistant-bf16★ 13

mlx safetensors gemma4_assistant speculative-decoding mtp gemma

Qwen-AgentWorld-35B-A3B-oQ4★ 13

mlx safetensors qwen3_5_moe qwen world-model agent

Local-LLM-Hardware-Checker★ 13

static region:us

llama-agent★ 12

Fun project to run your own LLM chat bot using llama.cpp

koog-edge★ 12

A bridge between JetBrains Koog framework and on-device LLM inference engines

DeepSeek-V2-Lite-Chat-IMat-GGUF★ 12

gguf quantized GGUF imatrix quantization imat

Taiwan-LLM-7B-v2.0.1-chat-GGUF★ 12

transformers gguf text-generation zh license:apache-2.0 region:us

Lemur-70B-Chat-v1-GGUF★ 12

transformers gguf llama text-generation code text-generation-inference

llama-2-7b-langchain-chat-GGUF★ 12

gguf text-generation pt en es ru

Qwythos-9B-Claude-Mythos-5-1M-GGUF_Q4_K_M★ 12

docker region:us

drama_llama★ 12

Rusty inference library using various backends including llama.cpp

pollard-weights★ 12

A tiered-memory system design for workloads that don't fit in RAM: measure the working set, pin the hot tier, stream the cold tier from flash. Ships the residency calculator, measurement harnesses, and the build recipes behind it. Predictions validated against public benchmarks.

Foq★ 12

⚡ Foq — the FREE, local, open-source alternative to Jev. Typed System 1 decisions in ~25 ms — no waitlist, no cloud, no per-token cost. foq.fr

ClippyAI★ 11

ClippyAI is a multi-platform GUI application, which serves as an interface for Ollama.

dgx-spark★ 11

NVIDIA DGX Spark ressources

web-llm-embed★ 11

docker region:us

SuperQwen-AgentWorld-35B-A3B-abliterated-gguf-4bit★ 11

gguf qwen-agentworld abliterated 4-bit quantized llama.cpp

NativeLab★ 11

**A fully local, privacy-first LLM desktop application powered by llama.cpp**

Jack-Long-Agentic-27B-4bpw-16GB-VRAM★ 11

gguf jack-xml j-space agentic long-context qwen3.6

taardis-27b-full-ternary★ 11

gguf ternary quantization integer llama.cpp taardis

infercrate★ 11

InferCrate: self-hosted local LLM server powered by Ollama. Secure by default with auto-generated Bearer token auth. Native Ollama API and an OpenAI-compatible /v1 API subset. First-start model pre-pull, NVIDIA GPU (CUDA) acceleration, and persistent model storage. Docker deployment on CPU (amd64/arm64) or CUDA (amd64).

CandleMist★ 10

Fullstack chatbot built using Rust. Made using Candle, Leptos, Actix, Tokio and Tailwind. Uses quantized Mistral 7B Instruct v0.1 GGUF models.

Local-LLM-Chatbot-GUI-Ollama★ 10

Local LLM Chatbot GUI. Interacts with open-source models offline via Ollama. Built in Python with a sleek interface. / Interfaz grafica local para chatear offline con modelos open-source usando Ollama.

deepseek-chatbot★ 10

A step-by-step guide to setting up and running DeepSeek locally using Ollama, making it accessible via a browser as an offline chatbot.

ollama-chatbot★ 10

Chat bot based on Ollama

SharpAI★ 10

SharpAI is an embeddable embeddings, completions, and model management platform using llama.cpp via LlamaSharp, with a built-in webserver with Ollama and OpenAI APIs.

LMStud★ 10

Chat with GGUF LLMs using llama.cpp and a classic Windows Forms interface for minimal GUI bloat.

docker-ollama★ 10

Docker image to run an Ollama local LLM server. Secure by default, all API requests require a Bearer token (auto-generated on first start). OpenAI-compatible API. Supports first-start model pre-pull, NVIDIA GPU (CUDA) acceleration, and persistent model storage. Multi-arch: amd64, arm64.

llama-bro★ 10

High-performance Android SDK for on-device LLM inference (GGUF). Privacy-focused, offline-first, and powered by llama.cpp with a clean Kotlin Coroutines API.

prism★ 10

Programmable Responsive Interface for Smart Models

ligguf★ 10

Little gguf - a tiny, embeddable, fully functional direct quantized inference engine for llama/qwen models in standard gguf format; no dependencies

AI-Offline-Chat-ADV★ 10

Fully autonomous offline NLP AI (ELIZA) for M5Stack Cardputer ADV.

awesome-on-device-mobile-llms★ 10

Curated resource for mobile teams shipping on-device LLMs — runtime benchmarks, model picks, GDPR-friendly architecture, and real production use cases.

local-inference-optimizer-skill★ 10

Skill для Hermes / Claude Code / Codex-агентов: автоматический выбор и настройка LLM inference engine под железо, модель и workload — MLX, llama.cpp, ExLlama, vLLM, SGLang, TensorRT-LLM.

AI-HOMELAB★ 10

🇺🇦 AI-HomeLab: Домашні AI-Лабораторії в Україні — локальний ШІ, мультиагентні системи, блекаут-резилієнтність

openai-gpt-oss-20b★ 10

gradio region:us

TinyLlama-1.1B-Chat-v0.3-GPTQ★ 10

transformers safetensors llama text-generation en dataset:cerebras/SlimPajama-627B

Yi-1.5-9B-Chat-GGUF★ 10

gguf text-generation base_model:01-ai/Yi-1.5-9B-Chat base_model:quantized:01-ai/Yi-1.5-9B-Chat license:apache-2.0 endpoints_compatible

TinyLlama-1.1B-Chat-v0.6-GGUF★ 10

gguf ggml quantized q2_k q3_k_m q4_k_m

vpngrok★ 10

Run xAI's Grok CLI from any project, with all traffic routed through a VPN (Docker + gluetun, Mullvad by default)

litertlm-kmp★ 10

Kotlin Multiplatform engine for running Gemma LLMs on-device on Android via LiteRT-LM — stateful KV-cache chat sessions, resumable model management, function calling. Includes NativeLM, a private Local AI chat app. AGPL-3.0 / commercial.

Echo_Adapt_v5★ 10

"Local LLM agent framework in Rust — native tool-role support, tmux session persistence, two-tier summarization, and ShareGPT logging. Works with any OpenAI-compatible backend."

Lociant★ 10

An Android-native local AI runtime with MCP tools, phone capabilities, and OpenAI-compatible APIs.

dsh-harmonyos-pc★ 10

让 DeepSeek Harness(dsh) 在 HarmonyOS 完整跑起来:缓存优化对话预设 + 启动补丁 + 工具链 | Run DeepSeek Harness on HarmonyOS: cache-optimized agent presets, patches & tooling

xyntetik-runner★ 10

Single-binary GGUF model runtime in C — CPU/CUDA/Metal, OpenAI-compatible. Serves, scores, and trains LoRA directly through the quantized weights it deploys, with byte-reproducible adapters. Tool calls survive the token limit; sparse MoE and schema-constrained decoding included.

codegemma-7b-it-GGUF★ 10

transformers gguf text-generation license:gemma endpoints_compatible region:us

protoagent★ 10

A blazing-fast, local-first AI agent ecosystem. Orchestrate local LLMs with Python, run them in a native Rust terminal, or integrate directly into your editor via ACP.

LibLayaX★ 10

Fast local AI decisions inside your application. A shared library and C API built on laya.cpp that runs the Laya typed-decision model in-process: no server, no Python. Windows, Linux and macOS, x64 and ARM64, CPU or GPU. An open-source alternative to Jev

Adaptive-Cognitive-Budgeting-GPU-Free-LLMs★ 10

Adaptive Cognitive Budgeting (ACB) dynamically allocates context windows and CPU threads based on query complexity and host memory pressure to prevent catastrophic latency stalls in GPU-free LLM inference.

Local-AI★ 10

An open-source, on-device AI assistant for Android. Chat with GPT, Claude, Gemini or your own local models, keep long-term memory on your phone, and let the AI use real tools. No account, no tracking, no telemetry. Built with Kotlin and Material 3 Expressive.

flint★ 10

Flint: a local-first agentic workspace, a fork of Jan

speechless-codellama-34b-v2.0-GGUF★ 10

transformers gguf llama llama-2 code text-generation

FrugalAI★ 10

Evidence-first frugal AI: small language models and pipelines on an old laptop or Raspberry Pi, no cloud, no GPU. Measured benchmarks and recipes, with Indian-language focus.

llama-compose★ 9

Kotlin Multiplatform application showcase with llama.cpp on device LLM inference & Koog.ai agent

tg-local-llm★ 9

Run local LLMs powered up by tools in Telegram Messenger

llmedge-examples★ 9

Examples using the llmedge library

lm-webui★ 9

Sovereign Local AI Workspace & LLM Runtime Manager. Easily Run GGUF, MLX, OpenAI, Gemini, etc. Under One Unified Interface ✨

NuraOS★ 9

A tiny, headless, AI-native operating system: a minimal Linux + musl/BusyBox appliance that boots straight into an on-device LLM agent over a local HTTP API and serial console. Local-first inference with a Rust agent core, Go gateway, and per-service cgroup/seccomp/Landlock isolation.

Qwen2.5-Coder-14B-Instruct-MLX-4bit★ 9

mlx safetensors qwen2 code codeqwen chat

Llama3-ChatQA-1.5-70B-GGUF★ 9

gguf nvidia chatqa-1.5 chatqa llama-3 pytorch

Dagger-Qwen3.6-27B-MLX★ 9

mlx safetensors qwen3_5 qwen3_6 token-efficient efficient-thinking

GLM-4.7-Flash-NEO-CODE-Imatrix-MAX-GGUF★ 9

gguf GLM 4.7 Flash thinking reasoning NEO Imatrix MAX Quants

moziAI-35B-A3B-MOE-MTP★ 9

MoziAI-35B: Local open-source financial AI LLM. 35B MoE model compressed to 15.5GB via MoziSmartBit quantization. 256K context, vision, tool calling, 140+ tok/s on consumer GPU. Based on Qwen3.6-35B-A3B.

LowRAM-AI-Compiler★ 9

A lightweight, memory-budgeted GGUF inference runtime and developer playground for running small language models on constrained devices.

node0-seed★ 9

The Seed: the smallest version of Node0, rung by rung, built on a bench with an agent: pages, costs, pitfalls, measurements and the configuration as it ran. Mirror of the Forgejo origin.

aerus-ezai★ 9

EZ AI — Local AI made easy: Windows command center for local AI apps, model library, and coding agents.

ext-infer★ 9

PHP extension for local inference from pre-trained models.

Luwi★ 9

DSH for Android, built on LittleWhale (dsh ported to Android by Miuzarte)

LFM3-1.7B-Claude-Code-GGUF★ 9

gguf lfm3 claude stepbystep agent toolcall

DSH-forAndroid★ 9

在 Android 本地运行 DeepSeek Harness,支持插件、悬浮对话与 Root / Shizuku。Run DeepSeek Harness locally on Android.

Peridot★ 8

Peridot v1.6.0 [STABLE] | Sovereign AI Kernel. Offline, GPU accelerated Qwen framework: a 27B model fully on an 8GB GPU at 23.6 tok/s. Sandboxed skills & plugins, model tool use, opt in web search, MCP bridge for private data jobs from cloud agents, ZAT-SCS preemption, RAG. Zero data exfiltration. Engineered by uncoalesced.

mlx-chainlit-llama3★ 8

Create a chatbot in less than 2 minutes with MLX Server, Chainlit and Llama3

ollamarama★ 8

Terminal-based AI chatbot with infinite personalities, using local LLMs via ollama

ollapa★ 8

An Ollama client built with Svelte 5 and SvelteKit. Chat completely local and client-side with a friendly interface!

DeepSeek-R1_distilled★ 8

GUI Running DeepSeek's reasoning model distilled from DeepSeek-R1 based on Llama and Qwen.

node-red-flow-openai-api★ 8

Node-RED Flows for OpenAI API compatible endpoints calling llama.cpp

ai-local-agents★ 8

A collection of intelligent AI agents built using Ollama, LangChain, and local LLMs, including chatbots, voice assistants, web scrapers, document readers, and specialized tools. No API required. Fully autonomous and privacy-preserving.

lemonade-python-sdk★ 8

A production-grade Python SDK for the Lemonade LLM backend. Featuring auto-discovery, port scanning, and a low-overhead client architecture. Powering the Sorana AI workspace.

local-genai-lab★ 8

Local-first GenAI lab with a React frontend and Spring Boot backend, supporting Ollama, Amazon Bedrock, Hugging Face, RAG document retrieval, MCP-backed AWS tools, streaming chat, persistent sessions, and artifact/report workflows.

nvidia-ai-hub★ 8

NVIDIA AI Hub — One-click AI app launcher for NVIDIA DGX GPU. Install, run, and manage GPU-accelerated AI apps from a modern web UI.

dgx-spark-nemotron-super-agent★ 8

Nemotron-3-Super-120B Cogni-Brain on DGX Spark: 131K context, ~24 tok/s, 93/100 Tool-Eval, hardened agent stack.

mlx-moe★ 8

Run large Mixture-of-Experts LLMs that exceed system RAM on Apple Silicon by loading only router-selected experts from SSD with MLX. Includes OpenAI/Anthropic-compatible serving for local agentic coding.

HuggingClip★ 8

Run PaperClip On HuggingFace Space

local-ai★ 8

Self-hosted Open source AI platform with RAG, TTS, fully offline

IBM-Grok4-Ultra.Fast.Coder-1B-GGUF★ 8

gguf llama 1b text-generation local-llm llama-cpp

Qwen2.5-Coder-7B-Instruct-MLX-4bit★ 8

mlx safetensors qwen2 code codeqwen chat

DavidAU-Qwen3.5-9B-Claude-4.6-HighIQ-THINKING-HERETIC-UNCENSORED★ 8

gradio region:us

h2o-danube2-1.8b-chat-GGUF★ 8

transformers gguf gpt llm large language model h2o-llmstudio

Kimi-K3-REAP73-zh-code-MLX-mxfp4-q8★ 8

mlx safetensors kimi_k3 moe reap pruned

RigMatch★ 8

Speed dating for local AI models: find the best one your PC can actually run. Tests Ollama models on your own hardware.

h2o-danube2-1.8b-chat-GGUF★ 8

transformers gguf gpt llm large language model h2o-llmstudio

codegemma-2b-GGUF★ 8

transformers gguf text-generation license:gemma endpoints_compatible region:us

hermes-local-ai-stack★ 8

Field-tested local AI stack optimizations for HermesAgent, LM Studio, and Qwen 3 models

Rombos-Coder-V2.5-Qwen-14b-GGUF★ 8

gguf code qwen qwen-coder codeqwen text-generation

mithril★ 8

Lightweight standalone LLM inference engine. Single binary, no runtime deps. Serves GGUF models via Ollama/OpenAI/MCP APIs. Multi-agent orchestration, TUI, 21 built-in tools.

Qwen2.5-Coder-7B-Instruct-GGUF★ 8

transformers gguf code codeqwen chat qwen

More in Agent frameworks & infrastructure