Agent frameworks & infrastructure
Local LLM runtimes
1,401 agents · most starred first
Local LLM runtimes, page 6 of 15
MoziAI-35B: Local open-source financial AI LLM. 35B MoE model compressed to 15.5GB via MoziSmartBit quantization. 256K context, vision, tool calling, 140+ tok/s on consumer GPU. Based on Qwen3.6-35B-A3B.
Run local LLMs fully offline - chat, coding agent, image, music, video generation, knowledge/RAG, voice. Privacy-first, with GUI and CLI, no cloud required.
gguf glm codegeex thudm quantized GGUF
The harness that makes consumer-GPU models feel first-class. Local-first: no telemetry, no account, no cloud tier.
llama.cpp gguf glm4-moe-lite quantized text-generation conversational
gguf llama.cpp qwen3.5 chat creative-writing altworld
OAra Prometheus — an open-source AI agent daemon that runs on hardware you own.
A flexible, extensible AI agent backend built with NestJS—designed for running local, open-source LLMs (Llama, Gemma, Qwen, DeepSeek, etc.) via Docker Model Runner. Real-time streaming, Redis messaging, web search, and Postgres memory out of the box. No cloud APIs required!
3 use cases for gemma 3n that you can install in a few lines of your bash on your local machine or a small server
V-lang api wrapper for llm-inference chatllm.cpp
💬 A CLI app that lets you watch conversations between two LLMs (all running locally using Ollama)
On-device AI chat for Android: Gemma 4 (E2B) through Google LiteRT LM, GPU-accelerated and fully offline. Text, vision and audio, with an optional Gemini path. Kotlin + Compose.
The official repository of NELA
Chat with tool calling models. Sample tools included. Add new tools to your shed with ease. Runs on Ollama. All via Bash shell scripts.
🐆 Self-hosted AI workstation stack for @amd
Turn your Windows GPU into a private, low-latency LLM server. Docker-based, OpenAI-compatible API.
🧠 EdgeBrain — Lightweight edge AI inference engine. Run ML models locally with sub-100ms latency. No cloud. No API keys. Just fast, private AI.
A fully auto configured, self-hosted local AI & database stack on Debian WSL2.
Local AI (LLM) Ollama Tool and Extension. It's a hammer /approve /reject are your friends.
[WIP] super dope kit for simple local personal agent; connect your own llama.cpp, vllm backend
A turnkey, fully-local AI workstation engineered for the AMD Ryzen AI Max+ 395. LLM inference, voice, document parsing, browser automation, agents — all on-device.
Free offline AI chat for Windows. Llama 3.2 3B model. No API keys, no censorship, no internet required. 100% private.
Local AI platform: run LLMs, Stable Diffusion, Whisper, TTS, video & 3D generation offline — open-source Ollama + ComfyUI alternative in one binary.
gguf roleplay phi-3 conversational phi-3-mini character
mlx safetensors gemma4_assistant speculative-decoding mtp gemma
mlx safetensors qwen2 code codeqwen chat
gguf generated_from_trainer text-generation base_model:migtissera/Trinity-2-Codestral-22B-v0.2 base_model:quantized:migtissera/Trinity-2-Codestral-22B-v0.2 lice
gguf llama.cpp nvfp4 mtp speculative-decoding moe
gguf unsloth llama.cpp qwen3_5 code quantized
Swift-first toolkit for shipping on-device AI on iOS, macOS, visionOS, tvOS and watchOS — Core ML LLM runtime with SwiftUI prefabs, structured output, tool calling, RAG and speech.
gguf heretic uncensored decensored abliterated ara
Is Nanbeige 4.2 the Best Small AI Model? Tested - Local web studio and agent engine powered by Nanbeige 4.2-3B.
On-device LLM chat — run open models (Gemma, Phi, Llama) fully offline with document Q&A (RAG) and a tool-calling agent. Built with Flutter + llama.cpp. Private by design: no cloud, no telemetry.
Run and serve local LLMs everywhere .NET runs - desktop and the browser - on the SpawnDev.ILGPU.ML inference engine (GGUF models, KV-cache decode, WebGPU dispatch-plan capture/replay).
transformers gguf text-generation zh license:apache-2.0 region:us
Hardware-focused llama.cpp fork for Windows and dual AMD RDNA4 RX 9070 XT GPUs: ROCm/HIP + Vulkan backends, PyQt6 GUI (RDNA LLM Studio), MTP speculative decoding, FP8 attention, long-context Qwen3.8 benchmarks
FreeAI: Unified AI Inference Workstation — local LLMs, autonomous agents, full SDLC automation, security tools, and multi-provider routing. One self-hosted stack.
Six verbs for on-device AI in TypeScript. Edgewise runs language, vision, speech, judging, embedding, image and forecasting models where your code runs: in the browser through WebGPU or WebAssembly, and in Bun or Node through ONNX Runtime, with vgpu for WebGPU on the server.
An Local AI Engine
Local AI stack for an AMD RX 580 8 GB: OpenAI-compatible router with 5 text/vision models up to 35B MoE (Qwen3.6, Ornith 35B), SD 3.5 + SD 1.5, Whisper and an agent engine scored on hidden-grader tasks. Vulkan, one install script, measured benchmarks. Inspired by Strata.
MiroFish (Apple Silicon Edition) — 针对 Apple Silicon (M1/M2/M3/M4) 与 MLX 深度优化的群体智能引擎(推理提速约 11x)
A production-grade, privacy-first desktop AI workstation & UI for local LLMs interfacing directly with Ollama.
On-device LLM chat — run open models (Gemma, Phi, Llama) fully offline with document Q&A (RAG) and a tool-calling agent. Built with Flutter + llama.cpp. Private by design: no cloud, no telemetry.
A privacy-first VS Code extension that integrates with any local LLM through Ollama. Chat with AI models directly in your editor without sending data to the cloud.
Local AI workstation dashboard Neuralforge— manage LLMs, agents, RAG, Telegram AI bot with 14 meme personas & voice cloning, image/video/3D generation, and LoRA fine-tuning and SMM module from a single web UI. Runs entirely on your hardware.
A web app that uses LangChain and vLLM to run an interactive AI agent 🤖
Local-first AI inference platform for LLMs — private chat, RAG, defense-in-depth security (Daily/Bulbe, encryption at rest, 2FA), on your own hardware. SvelteKit + FastAPI
Local LLM chat inerface for OpenCloud with on-premise AI, full data sovereignty and enterprise-grade security.
Run DeerFlow on Huggingface Space for Free
Run weak/local LLMs on narrow workflows reliably: encode a workflow once at design time as a deterministic macro, then a small local model just routes to it at runtime. TypeScript SDK, Apache-2.0, BYO-model (OpenAI-compatible + Ollama).
static region:us
docker inference api openai-api compatible llama-cpp-python Llama-2-7b-chat-hf-function-calling-v2-GGUF gguf region:us
mlx safetensors qwen3 coding research deep thinking
transformers safetensors llama text-generation conversational en
transformers gguf deepseek_v2 text-generation custom_code base_model:deepseek-ai/DeepSeek-V2-Lite-Chat
gguf text-generation license:apache-2.0 endpoints_compatible region:us imatrix
transformers gguf nanbeige llm custom_code text-generation
transformers gguf deepseek_v2 text-generation custom_code base_model:deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct
transformers gguf gemma text-generation llama-cpp gguf-my-repo
gguf gemma4 claude-code codex opencode coding
transformers gguf world-model qwen3_5_moe base_model:Qwen/Qwen3.5-35B-A3B-Base image-text-to-text
transformers gguf llama-cpp gguf-my-repo node-llama-cpp text-generation
transformers safetensors qwen2 text-generation code codeqwen
gguf text-generation base_model:01-ai/Yi-Coder-1.5B-Chat base_model:quantized:01-ai/Yi-Coder-1.5B-Chat license:apache-2.0 endpoints_compatible
gguf nvidia chatqa-1.5 chatqa llama-3 pytorch
gguf nvidia chatqa-1.5 chatqa llama-3 pytorch
A privacy-first Android AI assistant that runs entirely offline using GGUF language models powered by llama.cpp. Download, switch, and chat with multiple on-device LLMs—no internet required for AI inference after the initial model download. Built with Kotlin, Jetpack Compose, MVVM, Clean Architecture, and JNI.
mlx gguf claude-code coding-agent tool-calling qwen3.5
transformers gguf code codeqwen chat qwen
transformers gguf llama.cpp text-generation ar en
gguf text-generation base_model:andrewzh/Absolute_Zero_Reasoner-Coder-7b base_model:quantized:andrewzh/Absolute_Zero_Reasoner-Coder-7b endpoints_compatible regi
gguf code qwen qwen-coder codeqwen text-generation
transformers.js onnx qwen2 text-generation conversational base_model:Qwen/Qwen2.5-Coder-3B-Instruct
transformers gguf text-generation license:gemma endpoints_compatible region:us
gguf quantized GGUF quantization imat imatrix
transformers gguf mistral quantized 2-bit 3-bit
llama-cpp-python gguf llama tool-calling function-calling ai-agents
This 4B Local AI Scores 96.9% on Coding! (NeoHorse Agent Setup) - Complete setup guide and python implementation for running NeoHorse 1-4B local autonomous agent with Ollama.
mlx safetensors qwen2 code codeqwen chat
A capable 27B AI agent and coding assistant that runs entirely on your GPU: 77 tok/s code, 700 tok/s prefill and 128K context on one RTX 3060 12 GB. v0.1.5 ships the L0xRE-V2-27b-Low model. Linux and Windows, RTX 30/40/50 series.
Resident local AI inference daemon & multimodal studio in Rust. Subagent model orchestration (DMT), 3D mesh, image, speech & LLMs on consumer GPUs.
Light weight Ollama manger: setup and securely share your local Ollama models.
Local-first AI chat app for on-device LLMs, private RAG, and mobile edge AI workflows.
֎ Simple, privacy-first web UI for local LLMs (Ollama, LM Studio, OpenAI-compatible APIs)
Ollama model provider and utilities for adk-go
End To End Gen AI App Using DeepSeek-R1 With Langchain And Ollama. Released in January 2025, this model is based on DeepSeek-V3 and is focused on advanced reasoning tasks directly competing with OpenAI's o1 model in performance, while maintaining a significantly lower cost structure
Langchain Ollama Streaming example implemented in the flask
Teaching how to run AI locally.
Front End web App for Ollama
miniGPT built on Apple's MLX array framework
Most simple and minimal code to run an LLM chatbot from HuggingFace hub with OpenVINO
✅🦙 Kllama: The Local & Private Chatbot :dependabot:
A chat bot using various LLM models. It runs locally using your own GPU and/or CPU.
OllamaBot: Um assistente virtual inteligente, baseado em contêiner, capaz de interagir naturalmente em uma variedade de contextos.
Simple MLX + Streamlit Chatbot
Private LLModel GUI Chat allows users to interact with a local large language model (LLM) using the Llama library and GGUF models.
Adaption of KoboldCPP with the goal to add missing core features
A pure C# LLM inference engine built from scratch — no Python, no llama.cpp bindings, no ONNX Runtime. SharpInfer loads GGUF and Safetensors models directly, dequantizes weights in managed code, and runs the full transformer forward pass natively on .NET 8.
Welcome to the Mixtral 8x7B offloading demo repository! This project aims to demonstrate the seamless execution of Mixtral-8x7B models on Colab or consumer desktops.