Agent frameworks & infrastructure

Local LLM runtimes

1,401 agents · most starred first

Local LLM runtimes, page 6 of 15

moziAI-35B-A3B-MOE-MTP-Uncensored★ 8

MoziAI-35B: Local open-source financial AI LLM. 35B MoE model compressed to 15.5GB via MoziSmartBit quantization. 256K context, vision, tool calling, 140+ tok/s on consumer GPU. Based on Qwen3.6-35B-A3B.

localm★ 8

Run local LLMs fully offline - chat, coding agent, image, music, video generation, knowledge/RAG, voice. Privacy-first, with GUI and CLI, no cloud required.

codegeex4-all-9b-IMat-GGUF★ 8

gguf glm codegeex thudm quantized GGUF

shun★ 8

The harness that makes consumer-GPU models feel first-class. Local-first: no telemetry, no account, no cloud tier.

GLM-4.7-Flash-Coder-GGUF★ 8

llama.cpp gguf glm4-moe-lite quantized text-generation conversational

Astrea-R8-Chat-9B-GGUF★ 8

gguf llama.cpp qwen3.5 chat creative-writing altworld

Prometheus★ 8

OAra Prometheus — an open-source AI agent daemon that runs on hardware you own.

local-agent-docker-model-runner★ 7

A flexible, extensible AI agent backend built with NestJS—designed for running local, open-source LLMs (Llama, Gemma, Qwen, DeepSeek, etc.) via Docker Model Runner. Real-time streaming, Redis messaging, web search, and Postgres memory out of the box. No cloud APIs required!

gemma3n_use_case★ 7

3 use cases for gemma 3n that you can install in a few lines of your bash on your local machine or a small server

chatllm.v★ 7

V-lang api wrapper for llm-inference chatllm.cpp

InfiniChat★ 7

💬 A CLI app that lets you watch conversations between two LLMs (all running locally using Ollama)

MEDHA★ 7

On-device AI chat for Android: Gemma 4 (E2B) through Google LiteRT LM, GPU-accelerated and fully offline. Text, vision and audio, with an optional Gemini path. Kotlin + Compose.

nela★ 7

The official repository of NELA

ollama-bash-toolshed★ 7

Chat with tool calling models. Sample tools included. Add new tools to your shed with ease. Runs on Ollama. All via Bash shell scripts.

panther-minor★ 7

🐆 Self-hosted AI workstation stack for @amd

llm-windows-server★ 7

Turn your Windows GPU into a private, low-latency LLM server. Docker-based, OpenAI-compatible API.

EdgeBrain★ 7

🧠 EdgeBrain — Lightweight edge AI inference engine. Run ML models locally with sub-100ms latency. No cloud. No API keys. Just fast, private AI.

wsl-assistant★ 7

A fully auto configured, self-hosted local AI & database stack on Debian WSL2.

lean-ai★ 7

Local AI (LLM) Ollama Tool and Extension. It's a hammer /approve /reject are your friends.

wavecat-sdk★ 7

[WIP] super dope kit for simple local personal agent; connect your own llama.cpp, vllm backend

TitanVault★ 7

A turnkey, fully-local AI workstation engineered for the AMD Ryzen AI Max+ 395. LLM inference, voice, document parsing, browser automation, agents — all on-device.

FreeAI-Desktop★ 7

Free offline AI chat for Windows. Llama 3.2 3B model. No API keys, no censorship, no internet required. 100% private.

Vortelio★ 7

Local AI platform: run LLMs, Stable Diffusion, Whisper, TTS, video & 3D generation offline — open-source Ollama + ComfyUI alternative in one binary.

luvGPT_phi3-uncensored-chat-GGUF★ 7

gguf roleplay phi-3 conversational phi-3-mini character

gemma-4-E2B-it-assistant-bf16★ 7

mlx safetensors gemma4_assistant speculative-decoding mtp gemma

Qwen2.5-Coder-32B-Instruct-MLX-4bit★ 7

mlx safetensors qwen2 code codeqwen chat

Trinity-2-Codestral-22B-v0.2-GGUF★ 7

gguf generated_from_trainer text-generation base_model:migtissera/Trinity-2-Codestral-22B-v0.2 base_model:quantized:migtissera/Trinity-2-Codestral-22B-v0.2 lice

Agents-A1-MTP-GGUF★ 7

gguf llama.cpp nvfp4 mtp speculative-decoding moe

Qwimi-3.6-27B-Coder-MTP-GGUF★ 7

gguf unsloth llama.cpp qwen3_5 code quantized

LocalAIKit★ 7

Swift-first toolkit for shipping on-device AI on iOS, macOS, visionOS, tvOS and watchOS — Core ML LLM runtime with SwiftUI prefabs, structured output, tool calling, RAG and speech.

Agents-A1-4B-Heretic-ARA-Refusals8-GGUF★ 7

gguf heretic uncensored decensored abliterated ara

RayCodes_Nanbeige_4.2★ 7

Is Nanbeige 4.2 the Best Small AI Model? Tested - Local web studio and agent engine powered by Nanbeige 4.2-3B.

loxi_llm★ 7

On-device LLM chat — run open models (Gemma, Phi, Llama) fully offline with document Q&A (RAG) and a tool-calling agent. Built with Flutter + llama.cpp. Private by design: no cloud, no telemetry.

SpawnDev.AI★ 7

Run and serve local LLMs everywhere .NET runs - desktop and the browser - on the SpawnDev.ILGPU.ML inference engine (GGUF models, KV-cache decode, WebGPU dispatch-plan capture/replay).

Taiwan-LLM-7B-v2.1-chat-GGUF★ 7

transformers gguf text-generation zh license:apache-2.0 region:us

llama.cpp-rdna-lab★ 7

Hardware-focused llama.cpp fork for Windows and dual AMD RDNA4 RX 9070 XT GPUs: ROCm/HIP + Vulkan backends, PyQt6 GUI (RDNA LLM Studio), MTP speculative decoding, FP8 attention, long-context Qwen3.8 benchmarks

FreeAI_AI_Inference_Workstation★ 7

FreeAI: Unified AI Inference Workstation — local LLMs, autonomous agents, full SDLC automation, security tools, and multi-provider routing. One self-hosted stack.

edgewise★ 7

Six verbs for on-device AI in TypeScript. Edgewise runs language, vision, speech, judging, embedding, image and forecasting models where your code runs: in the browser through WebGPU or WebAssembly, and in Bun or Node through ONNX Runtime, with vgpu for WebGPU on the server.

AJAN★ 7

An Local AI Engine

polaris-local-ai★ 7

Local AI stack for an AMD RX 580 8 GB: OpenAI-compatible router with 5 text/vision models up to 35B MoE (Qwen3.6, Ornith 35B), SD 3.5 + SD 1.5, Whisper and an agent engine scored on hidden-grader tasks. Vulkan, one install script, measured benchmarks. Inspired by Strata.

mirrorfish_appleSilicon★ 7

MiroFish (Apple Silicon Edition) — 针对 Apple Silicon (M1/M2/M3/M4) 与 MLX 深度优化的群体智能引擎(推理提速约 11x)

LocalLLMMind★ 7

A production-grade, privacy-first desktop AI workstation & UI for local LLMs interfacing directly with Ollama.

loki_llm★ 7

On-device LLM chat — run open models (Gemma, Phi, Llama) fully offline with document Q&A (RAG) and a tool-calling agent. Built with Flutter + llama.cpp. Private by design: no cloud, no telemetry.

OmniChat-VSCode-Extension★ 6

A privacy-first VS Code extension that integrates with any local LLM through Ollama. Chat with AI models directly in your editor without sending data to the cloud.

neuralforge★ 6

Local AI workstation dashboard Neuralforge— manage LLMs, agents, RAG, Telegram AI bot with 14 meme personas & voice cloning, image/video/3D generation, and LoRA fine-tuning and SMM module from a single web UI. Runs entirely on your hardware.

local-vllm-agent★ 6

A web app that uses LangChain and vLLM to run an interactive AI agent 🤖

opti-oignon★ 6

Local-first AI inference platform for LLMs — private chat, RAG, defense-in-depth security (Daily/Bulbe, encryption at rest, 2FA), on your own hardware. SvelteKit + FastAPI

local-llm-opencloud★ 6

Local LLM chat inerface for OpenCloud with on-premise AI, full data sovereignty and enterprise-grade security.

HuggingFlow★ 6

Run DeerFlow on Huggingface Space for Free

core★ 6

Run weak/local LLMs on narrow workflows reliably: encode a workflow once at design time as a deterministic macro, then a small local model just routes to it at runtime. TypeScript SDK, Apache-2.0, BYO-model (OpenAI-compatible + Ollama).

Web-LLM-Mistral-7B-OpenOrca★ 6

static region:us

Llama-2-7b-chat-hf-function-calling-v2-GGUF★ 6

docker inference api openai-api compatible llama-cpp-python Llama-2-7b-chat-hf-function-calling-v2-GGUF gguf region:us

Qwen3-4B-Agent-F32-dwq4-mlx★ 6

mlx safetensors qwen3 coding research deep thinking

TinyLlama-1.1B-Chat-v1.0-AWQ★ 6

transformers safetensors llama text-generation conversational en

DeepSeek-V2-Lite-Chat-GGUF★ 6

transformers gguf deepseek_v2 text-generation custom_code base_model:deepseek-ai/DeepSeek-V2-Lite-Chat

Yi-1.5-34B-Chat-16K-GGUF★ 6

gguf text-generation license:apache-2.0 endpoints_compatible region:us imatrix

Nanbeige-16B-Chat-32K-GGUF★ 6

transformers gguf nanbeige llm custom_code text-generation

DeepSeek-Coder-V2-Lite-Instruct-GGUF★ 6

transformers gguf deepseek_v2 text-generation custom_code base_model:deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct

gemma-2b-it-Q4_K_M-GGUF★ 6

transformers gguf gemma text-generation llama-cpp gguf-my-repo

gemma4-26b-claude-coder-GGUF★ 6

gguf gemma4 claude-code codex opencode coding

Qwen-AgentWorld-35B-A3B-GGUF★ 6

transformers gguf world-model qwen3_5_moe base_model:Qwen/Qwen3.5-35B-A3B-Base image-text-to-text

Qwen3-Coder-30B-A3B-Instruct-Q4_K_M-GGUF★ 6

transformers gguf llama-cpp gguf-my-repo node-llama-cpp text-generation

Qwen2.5-Coder-3B-Instruct-4bit★ 6

transformers safetensors qwen2 text-generation code codeqwen

Yi-Coder-1.5B-Chat-GGUF★ 6

gguf text-generation base_model:01-ai/Yi-Coder-1.5B-Chat base_model:quantized:01-ai/Yi-Coder-1.5B-Chat license:apache-2.0 endpoints_compatible

Llama3-ChatQA-1.5-8B-GGUF★ 6

gguf nvidia chatqa-1.5 chatqa llama-3 pytorch

Llama3-ChatQA-1.5-70B-GGUF★ 6

gguf nvidia chatqa-1.5 chatqa llama-3 pytorch

offline-ai-assistant★ 6

A privacy-first Android AI assistant that runs entirely offline using GGUF language models powered by llama.cpp. Download, switch, and chat with multiple on-device LLMs—no internet required for AI inference after the initial model download. Built with Kotlin, Jetpack Compose, MVVM, Clean Architecture, and JNI.

qwen35-claude-coder★ 6

mlx gguf claude-code coding-agent tool-calling qwen3.5

Qwen2.5-Coder-7B-Instruct-Q4_K_M-GGUF★ 6

transformers gguf code codeqwen chat qwen

Jais-2-8B-Chat-GGUF★ 6

transformers gguf llama.cpp text-generation ar en

andrewzh_Absolute_Zero_Reasoner-Coder-7b-GGUF★ 6

gguf text-generation base_model:andrewzh/Absolute_Zero_Reasoner-Coder-7b base_model:quantized:andrewzh/Absolute_Zero_Reasoner-Coder-7b endpoints_compatible regi

Qwen2.5-Coder-0.5B-GGUF★ 6

gguf code qwen qwen-coder codeqwen text-generation

Qwen2.5-Coder-3B-Instruct★ 6

transformers.js onnx qwen2 text-generation conversational base_model:Qwen/Qwen2.5-Coder-3B-Instruct

codegemma-1.1-7b-it-GGUF★ 6

transformers gguf text-generation license:gemma endpoints_compatible region:us

DeepSeek-Coder-V2-Instruct-IMat-GGUF★ 6

gguf quantized GGUF quantization imat imatrix

BASH-Coder-Mistral-7B-Mistral-7B-Instruct-v0.2-slerp-GGUF★ 6

transformers gguf mistral quantized 2-bit 3-bit

yantra-1b-agent★ 6

llama-cpp-python gguf llama tool-calling function-calling ai-agents

RayCodes_NeoHorse★ 6

This 4B Local AI Scores 96.9% on Coding! (NeoHorse Agent Setup) - Complete setup guide and python implementation for running NeoHorse 1-4B local autonomous agent with Ollama.

Qwen2.5-Coder-7B-Instruct-MLX★ 6

mlx safetensors qwen2 code codeqwen chat

L0xRE-BeeLLama-Low★ 6

A capable 27B AI agent and coding assistant that runs entirely on your GPU: 77 tok/s code, 700 tok/s prefill and 128K context on one RTX 3060 12 GB. v0.1.5 ships the L0xRE-V2-27b-Low model. Linux and Windows, RTX 30/40/50 series.

DisposAI★ 6

Resident local AI inference daemon & multimodal studio in Rust. Subagent model orchestration (DMT), 3D mesh, image, speech & LLMs on consumer GPUs.

Llameros★ 6

Light weight Ollama manger: setup and securely share your local Ollama models.

core-app★ 5

Local-first AI chat app for on-device LLMs, private RAG, and mobile edge AI workflows.

localchat★ 5

֎ Simple, privacy-first web UI for local LLMs (Ollama, LM Studio, OpenAI-compatible APIs)

adk-go-ollama★ 5

Ollama model provider and utilities for adk-go

DeepSeek-R1-demo★ 5

End To End Gen AI App Using DeepSeek-R1 With Langchain And Ollama. Released in January 2025, this model is based on DeepSeek-V3 and is focused on advanced reasoning tasks directly competing with OpenAI's o1 model in performance, while maintaining a significantly lower cost structure

Run-Llm-on-Local-Ollama★ 5
langchain-ollama-stream★ 5

Langchain Ollama Streaming example implemented in the flask

keeping-your-intelligence-local★ 5

Teaching how to run AI locally.

Komli★ 5

Front End web App for Ollama

honeycrisp★ 5

miniGPT built on Apple's MLX array framework

openvino-llm-minimal-code★ 5

Most simple and minimal code to run an LLM chatbot from HuggingFace hub with OpenVINO

kllama★ 5

✅🦙 Kllama: The Local & Private Chatbot :dependabot:

bottled-ai★ 5

A chat bot using various LLM models. It runs locally using your own GPU and/or CPU.

ollama-bot★ 5

OllamaBot: Um assistente virtual inteligente, baseado em contêiner, capaz de interagir naturalmente em uma variedade de contextos.

streamlx★ 5

Simple MLX + Streamlit Chatbot

LLModel★ 5

Private LLModel GUI Chat allows users to interact with a local large language model (LLM) using the Llama library and GGUF models.

KoboldCS★ 5

Adaption of KoboldCPP with the goal to add missing core features

SharpInfer★ 5

A pure C# LLM inference engine built from scratch — no Python, no llama.cpp bindings, no ONNX Runtime. SharpInfer loads GGUF and Safetensors models directly, dequantizes weights in managed code, and runs the full transformer forward pass natively on .NET 8.

ColabMixtral★ 5

Welcome to the Mixtral 8x7B offloading demo repository! This project aims to demonstrate the seamless execution of Mixtral-8x7B models on Colab or consumer desktops.

More in Agent frameworks & infrastructure