Agent frameworks & infrastructure

Observability & evaluation

Observability and evaluation projects trace, log, and evaluate LLM and agent runs so teams can see what happened and whether it worked. Langfuse combines tracing with evals, while promptfoo focuses on testing prompts and agents, including red teaming and model comparisons.

5,667 agents · most starred first

Top 100 of 5,667

posthog★ 40,986

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

langfuse★ 37,668

🪢 Open source AI engineering platform: AI Gateway, agent observability, evals, and prompt/skill management, built on OpenTelemetry. One open platform for every AI call and every agent. Self-host or use Langfuse Cloud.

signoz★ 32,690

SigNoz is an open-source, OpenTelemetry-native observability platform for your team and their AI agents. Get logs, metrics, and traces in one tool with features like APM, distributed tracing, log management, infra monitoring, etc. Combined with SigNoz MCP and a native AI teammate (in SigNoz Cloud) it helps you build more resilient apps.

mlflow★ 28,907

The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

promptfoo★ 27,485

Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.

opik★ 23,462

Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.

RagaAI-Catalyst★ 16,192

Python SDK for Agent AI Observability, Monitoring and Evaluation Framework. Includes features like agent, llm and tools tracing, debugging multi-agentic system, self-hosted dashboard and advanced analytics with timeline and execution graph view

ccusage★ 14,702

Analyze coding (agent) CLI token usage and costs from local data.

open_llm_leaderboard★ 14,165

docker leaderboard modality:text submission:automatic test:public language:english eval:code eval:math

phoenix★ 12,364

AI Observability & Evaluation

codeburn★ 11,872

Free, local tool to track AI coding token usage and cost across 37 tools and agents (Claude Code, Cursor, Codex, Gemini and more), by model, project, and task. npx codeburn

tensorzero★ 11,701

TensorZero is an open-source LLMOps platform that unifies an LLM gateway, observability, evaluation, optimization, and experimentation.

failproofai★ 9,883

Observability and enforcement for AI agent harnesses. Capture every run and runtime reliability with policy enforcement.

openllmetry★ 7,543

Open-source observability for your GenAI or LLM application, based on OpenTelemetry

opencompass★ 7,400

OpenCompass is an LLM evaluation platform, supporting a wide range of models (Llama3, Mistral, InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets.

harbor★ 7,286

Framework for evaluating and improving agents

greptimedb★ 6,825

The open-source observability database. One columnar engine for metrics, logs, and traces, on object storage.

chinese-llm-benchmark★ 6,522

非线智能 NoneLinear - ReLE评测:中文AI大模型能力评测(持续更新):目前已囊括374个大模型,覆盖chatgpt、gpt-5.4、谷歌gemini-3.1-pro、Claude-4.6、文心ERNIE-X1.1、ERNIE-5.0、qwen3.6-max、qwen3.6-plus、百川、讯飞星火、商汤senseChat等商用模型, 以及step3.5-flash、kimi-k2.6、ernie4.5、MiniMax-M2.7、deepseek-v4、Qwen3.6、llama4、智谱GLM-5.1、MiMo-V2、LongCat、gemma4、mistral等开源大模型。不仅提供排行榜,也提供规模超200万的大模型缺陷库!方便广大社区研究分析、改进大模型。

mission-control★ 6,511

Self-hosted control plane for AI agents: dispatch tasks, review runs, track spend, and operate OpenClaw, Claude Code, Codex, and other runtimes.

agentsview★ 6,489

Local-first session search, analytics, insights, and token use statistics for coding agents, supporting Claude Code, Codex, and more than 50 other agents.

helicone★ 6,291

🧊 Open source LLM observability platform. One line of code to monitor, evaluate, and experiment. YC W23 🍓

agentops★ 5,968

Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks including CrewAI, Agno, OpenAI Agents SDK, Langchain, Autogen, AG2, and CamelAI

giskard-oss★ 5,961

🐢 Open-Source Evaluation & Testing library for LLM Agents

tokscale★ 5,879

🛰️ Track token usage across AI coding agents from your terminal. 🏅 Global leaderboard with trillions of tokens tracked.

coze-loop★ 5,800

Next-generation AI Agent Optimization Platform: Cozeloop addresses challenges in AI agent development by providing full-lifecycle management capabilities from development, debugging, and evaluation to monitoring.

ToolBench★ 5,771

[ICLR'24 spotlight] An open platform for training, serving, and evaluating large language model for tool learning.

Kiln★ 5,285

Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.

argilla★ 5,208

Argilla is a collaboration tool for AI engineers and domain experts to build high-quality datasets

langwatch★ 5,047

Open-Source Agent Observability & Evals for AI Platform teams. Trace, test, route and govern every agent & coding assistant in the company.

latitude-llm★ 4,757

Open-source observability for AI agents. Find where your agents fail, dispatch your coding agent to fix it, and verify the fix against real traces.

logfire★ 4,561

AI observability platform for production LLM and agent systems.

VLMEvalKit★ 4,498

Open-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks

SkillEvaluator★ 4,204

Multi-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.

AgentBench★ 3,828

A Comprehensive Benchmark to Evaluate LLMs as Agents (ICLR'24)

evalscope★ 3,715

A streamlined and customizable framework for efficient large model (LLM, VLM, AIGC) evaluation and performance benchmarking.

trulens★ 3,661

Evaluation and Tracking for LLM Experiments and AI Agents

lmnr★ 3,513

Laminar - open-source observability platform purpose-built for AI agents. YC S24.

claude-tap★ 3,355

Intercept and inspect Coding Agent API traffic from Claude Code, Codex CLI, Gemini CLI, Cursor CLI, OpenCode, Kimi/Kimi Code, Pi, and Hermes in a local trace viewer.

llm-as-a-verifier★ 3,354

LLM-as-a-Verifier is a general-purpose framework that provides fine-grained feedback for any agent without requiring additional training. It achieves SOTA performance across coding, robotics, and medical agentic benchmarks.

openlit★ 3,091

Same agent, Smaller bill. The tools and infrastructure to get more work from your AI. Fewer tokens. Less compute. Lower cost.

AssetOpsBench★ 2,455

AssetOpsBench - Industry 4.0: A unified benchmark and framework for building, orchestrating, and evaluating domain-specific AI agents for Industry 4.0 asset operations and maintenance, with 460+ scenarios, 5 specialist agents (IoT, FMSR, TSFM, Work Order,...), and multi-agent orchestration blueprints (MetaAgent, AgentHive) over MCP.

parseable★ 2,431

Parseable is an open source, unified infrastructure observability platform built in Rust on a data lake architecture. It tracks logs, metrics, traces, and events across apps, agents, and systems, reducing storage costs by up to 90% through columnar telemetry compression.

future-agi★ 2,288

Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.

llm-space★ 2,143

A desktop app to prototype agent ideas, inspect every harness step, replay failures, and evaluate performance, all in one place. Local-first, cloud-ready for managed agents.

InferenceX★ 2,070

Open Source AI Accelerator Research Platform Standard / 开源推理研究平台

tokencost★ 2,004

Easy token price estimates for 400+ LLMs. TokenOps.

OpenClaw-bot-review★ 1,951

A lightweight web dashboard for viewing all your OpenClaw Bots/Agents/Models/Sessions status at a glance.

scaphandre★ 1,941

⚡ Energy consumption metrology agent. Let "scaph" dive and bring back the metrics that will help you make your systems and applications more sustainable !

skillsbench★ 1,930

SkillsBench evaluates how well skills work and how effective agents are at using them.

deep-swe★ 1,921

Measuring frontier coding agents on original, long-horizon engineering tasks

evals-skills★ 1,826

Skills that guide AI coding agents to help you build product-specific AI evals.

agent-flow★ 1,802

Real-time visualization of Claude Code agent orchestration — see your agents think, branch, and coordinate as they work.

mle-bench★ 1,801

MLE-bench is a benchmark for measuring how well AI agents perform at machine learning engineering

meta-harness★ 1,791

Official code for Meta-Harness (2603.28052)

SkyAPM-dotnet★ 1,657

The .NET/.NET Core instrument agent for Apache SkyWalking

Observal★ 1,588

Observal is a self hosted Docker Hub for human-in-the-loop agents with built in observability and AI insights

Wake★ 1,580

All your AI agent sessions in one place — browse, search, resume. Rust + GPUI.

Sentry★ 1,575

Learning to Recover from LLM Agent Failures at Test Time

claude-code-hooks-multi-agent-observability★ 1,571

Real-time monitoring for Claude Code agents through simple hook event tracking.

skill-up★ 1,508

Evaluate Agent Skills and environments. Evolve Skills with evidence.

chatgpt-comparison-detection★ 1,507

Human ChatGPT Comparison Corpus (HC3), Detectors, and more! 🔥

superlog★ 1,478

Open-source observability tool that uses AI agents to self-heal your software

traceroot★ 1,435

traceroot: the open-source platform that continuously improves your ai agents.

skill★ 1,384

PinchBench is a benchmarking system for evaluating LLM models as OpenClaw coding agents. Made with 🦀 by the humans at https://kilo.ai

openinference★ 1,331

OpenTelemetry Instrumentation for AI Observability

ClawBench★ 1,319

Open-source benchmark for browser AI agents on daily tasks.

intellagent★ 1,266

A framework for comprehensive diagnosis and optimization of agents using simulated, realistic synthetic interactions

any-agent★ 1,260

A single interface to use and evaluate different agent frameworks

langtrace★ 1,235

Langtrace 🔍 is an open-source, Open Telemetry based end-to-end observability tool for LLM applications, providing real-time tracing, evaluations and metrics for popular LLMs, LLM frameworks, vectorDBs and more.. Integrate using Typescript, Python. 🚀💻📊

uqlm★ 1,215

[JMLR 2026] UQLM: A Python package for LLM hallucination detection using uncertainty quantification

Claude-Code-Agent-Monitor★ 1,195

🚀 A real-time monitoring dashboard for Claude Code & Codex, built with SQLite3, Node.js, Express, React, Vite, TailwindCSS, & WebSockets. It tracks sessions, agent activity, tool usage, and subagent orchestration, providing live analytics, a Kanban status board, status notifications, a cute buddy, & an interactive web UI/MacOS/Windows native app.

Tracely-ai★ 1,122

Trace-native CI/CD for AI agents — production failures become regression tests that block the PR. Auto-detect, cluster, freeze into hermetic cases, replay in CI for $0.

OpenJudge★ 1,120

OpenJudge: A Unified Framework for Holistic Evaluation and Quality Rewards

terminal-bench★ 1,105

Measuring and evolving with the frontier of agent work

auto-evaluator★ 1,104

Evaluation tool for LLM QA chains

langsmith-sdk★ 1,081

This repo was archived on Oct 16, 2026. Python SDK moved to langchain-ai/langsmith-python and JS SDK to langchain-ai/langsmith-javascript.

judgeval★ 1,073

The Continuous-Improvement Stack for Agents. Our environment data and evals power agent improvement and monitoring.

workshop★ 1,063

Give your coding agent the power to write and run agent evals.

awesome-evals★ 1,046

A curated, non-BS library of the best resources for building and evaluating AI agents — papers, blogs, talks, tools, benchmarks. Maintained by BenchFlow.

agenttrail★ 1,045

Local observability for AI coding agents. Agenttrail Map shows your project as a live map of what each agent is working on, in the browser or natively in Herdr.

OrcaReplay★ 1,041

OrcaReplay — Time travel for AI agents. Record, replay, fork, and debug any agent run with any model. Built by the OrcaRouter.ai team.

DeepSeek-V4-J-Space-Capability-Realization-Report★ 1,035

DeepSeek V4 × J-Space capability realization report — benchmark evidence that J-Space reduces capability-realization loss on DeepSeek V4.

aimock★ 1,028

Mock everything your AI app talks to — LLM APIs, MCP, A2A, AG-UI, vector DBs, search. One package, one port, zero dependencies.

llm.report★ 1,018

📊 llm.report is an open-source logging and analytics platform for OpenAI: Log your ChatGPT API requests, analyze costs, and improve your prompts.

GLM-5.3-Flash-J-Space-Capability-Realization-Report★ 1,016

GLM-5.3-Flash × J-Space capability realization — benchmark presentation of the J-Space Cognition Suite

ccglass★ 989

See what your coding agent (Claude Code, Codex, Kimi) sends to the model — local proxy + web dashboard

terminal-bench-science★ 951

Terminal-Bench-Science: Evaluating AI agents on research workflows across scientific domains

claw-eval★ 946

Claw-Eval is an evaluation harness for evaluating LLM as agents. All tasks verified by humans.

hermes-hud★ 937

TUI consciousness monitor for Hermes

TokenTracker★ 927

Track token usage across 25 AI coding tools — Claude Code, Codex, Cursor, Gemini, Kiro, OpenCode, Antigravity, Copilot, Kimi, CodeBuddy, WorkBuddy, Grok, Kilo, Roo, Zed, Goose, Mimo, ZCode & more — local-first, zero-config, with a dashboard, macOS menu bar app, and desktop widgets.

leaderboard★ 908

static leaderboard benchmark agent-memory long-term-memory region:us

hermes-control-interface★ 901

A self-hosted web dashboard for the Hermes AI agent stack. Provides a browser-based terminal, file explorer, session overview, cron management, system metrics, and an agent status panel — all behind a single password gate.

crabwalk★ 872

🦀 Crabwalk 🦀 Real-time companion monitor for OpenClaw agents.

pandaprobe★ 867

open source agent engineering platform: traces, evals, and metrics to debug and improve your AI agents. Integrates with LangGraph, CrewAI, Claude Agent SDK, and more.

Tracely-ai★ 851

Trace-native CI/CD for AI agents: grade production traces, cluster failures, freeze them into replayable regression tests, and block PRs that regress. Self-hostable.

AgentCPM★ 835

An End-to-End Infrastructure for Training and Evaluating Various LLM Agents

ClawProBench★ 829

ClawProBench is a live-first benchmark harness for evaluating LLM agents in the OpenClaw runtime with deterministic grading and repeated-trial reliability.

agent-skills-eval★ 819

A test runner for agentskills.io-style AI agent skills

tokentap★ 812

Intercept LLM API traffic and visualize token usage in a real-time terminal dashboard. Track costs, debug prompts, and monitor context window usage across your AI development sessions.

vllora★ 810

Debug your AI agents

More in Agent frameworks & infrastructure