Agent frameworks & infrastructure

Observability & evaluation

5,667 agents ยท most starred first

Observability & evaluation, page 3 of 57

bananalyzerโ˜… 332

Open source AI Agent evaluation framework for web tasks ๐Ÿ’๐ŸŒ

hud-pythonโ˜… 329

RL environments + evals for AI agents. Define once, train anything.

tonic_validateโ˜… 327

Metrics to evaluate the quality of responses of your Retrieval Augmented Generation (RAG) applications.

MARBLEโ˜… 321

(ACL 2025 Main) Code for MultiAgentBench : Evaluating the Collaboration and Competition of LLM agents https://www.arxiv.org/pdf/2503.01935

kitaruโ˜… 321

Instill trust in your agent evals.

LAMMโ˜… 320

[NeurIPS 2023 Datasets and Benchmarks Track] LAMM: Multi-Modal Large Language Models and Applications as AI Agents

autoharnessโ˜… 319

๐Ÿ” Improve your agent harness autonomously overnight

agent-tally.nvimโ˜… 318

Track how your AI agents work, what they cost, and your agents heatmaps. Stop guessing your AI spend.

ML-Benchโ˜… 315

ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code (https://arxiv.org/abs/2311.09835)

T-Evalโ˜… 313

[ACL2024] T-Eval: Evaluating Tool Utilization Capability of Large Language Models Step by Step

AgentArkโ˜… 310

A General-Purpose Environment Framework for Scalable Multimodal Agent Evaluation and RL

agent-safety-eval-labโ˜… 309

Agent trace and tool-use safety evaluation lab.

archipelagoโ˜… 305

Harness for running and evaluating AI agents against RL environments

claw-lensโ˜… 305

Open-source observability dashboard for OpenClaw daemons โ€” cost analyti cs, live monitor ing, and de bugging tools, all running on your own machine.

ATaCโ˜… 302

Record and Replay AI Agent Trajectories.

elimination_gameโ˜… 298

A multi-player tournament benchmark that tests LLMs in social reasoning, strategy, and deception. Players engage in public and private conversations, form alliances, and vote to eliminate each other

ccxrayโ˜… 296

X-ray vision for AI agent sessions โ€” a transparent HTTP proxy and dashboard for Claude Code

beyondllmโ˜… 295

Build, evaluate and observe LLM apps

terminal-bench-3โ˜… 292

Measuring agents' ability to get work done on a computer

ToolQAโ˜… 288

ToolQA, a new dataset to evaluate the capabilities of LLMs in answering challenging questions with external tools. It offers two levels (easy/hard) across eight real-life scenarios.

agtopโ˜… 288

top-style TUI for monitoring AI coding agent sessions

reactbenchโ˜… 288

ReactBench is an evaluation for coding agents on realistic React work

bergenโ˜… 282

Benchmarking library for RAG

harbor-cookbookโ˜… 282

Realistic examples of building evals and optimizing agents with Harbor

lumlโ˜… 281

AI lifecycle platform where engineers and agents track experiments, train models, and ship to production.

chatbot-arenaโ˜… 280

static region:us

cognifyโ˜… 278

Multi-Faceted AI Agent and Workflow Autotuning. Automatically optimizes LangChain, LangGraph, DSPy programs for better quality, lower execution latency, and lower execution cost. Also has a simple agent/workflow framework

VisualAgentBenchโ˜… 278

Towards Large Multimodal Models as Visual Foundation Agents

BALROGโ˜… 277

Benchmarking Agentic LLM and VLM Reasoning On Games

graphsignalโ˜… 277

GPU Profiler for AI agents

Auto-GPT-Benchmarksโ˜… 276

A repo built for the purpose of benchmarking the performance of agents, regardless of how they are set up and how they work.

openclaw-mission-controlโ˜… 272

Mission Control is a simple, real-time web UI for monitoring agents and task workflows. Built with Convex and React, it provides a clean dashboard to track task state, agent activity, and live logs โ€” all synchronized instantly without polling or message queues.

floe-guardโ˜… 269

AI Initiative Ledger โ€” local spend meter and budget gate for AI tasks. Meters every vendor leg (LLM, speech, telephony, search, tools), hard-stops before the ceiling. MIT, no account.

Simreal-MLBenchโ˜… 269

[Public preview] Externally scored agentic ML research benchmark: 60 tasks, real competition ground truth. Open protocol, operated evaluation.

traceAIโ˜… 267

Open Source AI Tracing Framework built on Opentelemetry for AI Applications and Frameworks

loongsuite-pilotโ˜… 265

Local-first telemetry collector for AI coding agents โ€” unified OpenTelemetry events for Claude Code, Codex, Cursor and more. Token usage, cost, traces and security audit, exported anywhere.

benchllmโ˜… 263

Continuous Integration for LLM powered applications

langfairโ˜… 262

LangFair is a Python library for conducting use-case level LLM bias and fairness assessments

evalsโ˜… 261

A comprehensive evaluation framework for AI agents and LLM applications.

is-gpt-nerfedโ˜… 261

Shrinkflation detector for Codex

empiricaโ˜… 258

Make AI agents and AI workflows measurably reliable. Epistemic measurement, Noetic RAG, Sentinel gating, and grounded calibration for Claude Code and beyond

claude-carbonโ˜… 253

Track the carbon footprint of your Claude Code sessions, in the terminal and the Claude desktop app

langfuse-docsโ˜… 253

๐Ÿชข Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.

optimaizrโ˜… 253

Spend fewer tokens on the same work. A local CLI that finds wasted tokens in Claude Code, Codex, opencode and your own API calls.

confabulationsโ˜… 250

Hallucinations (Confabulations) Document-Based Benchmark for RAG. Includes human-verified questions and answers.

AgentBaseโ˜… 250

Multi-agent orchestrator for tracking and analyzing AI coding assistant conversations (Claude Code, Cursor, Windsurf)

graphsignal-profilerโ˜… 245

Graphsignal Profiler

open_pt_llm_leaderboardโ˜… 244

static leaderboard portuguese submission:closed test:public judge:auto modality:text eval:generation

evaโ˜… 243

A New End-to-end Framework for Evaluating Voice Agents

ts-benchโ˜… 242

Benchmark CLI for comparing AI coding agents on TypeScript workloads.

rl-plotterโ˜… 238

:sparkles: A plotter for reinforcement learning (RL)

arksimโ˜… 236

Find your agents errors be fore your real users do

caliperโ˜… 235

Run your real agent with and without your skills, MCPs, and rules. See which ones actually help, and what they cost in tokens. Supports Claude Code, Codex, Pi, Hermes, Antigravity, and OpenCode.

llm-bulls-and-cows-benchmarkโ˜… 234

A mini-framework for evaluating LLM performance on the Bulls and Cows number guessing game, supporting multiple LLM providers.

agent-studioโ˜… 233

[ICLR 2025] A trinity of environments, tools, and benchmarks for general virtual agents

low_bit_open_llm_leaderboardโ˜… 233

gradio leaderboard region:us

splitrailโ˜… 229

Fast, cross-platform, real-time token usage tracker and cost monitor for Claude Code / Codex CLI / Antigravity CLI / Qwen Code / Cline / Zoo Code / Kilo Code / GitHub Copilot / OpenCode / Pi Agent / Piebald.

empiricaโ˜… 226

Make AI agents and AI workflows measurably reliable. Epistemic measurement, Noetic RAG, Sentinel gating, and grounded calibration for Claude Code and beyond

agent-leaderboardโ˜… 225

Ranking LLMs on agentic tasks

scout_apm_rubyโ˜… 224

ScoutAPM Ruby Agent. Supports Rails, Sinatra, Grape, Rack, and many other frameworks

myclaw-benchโ˜… 222

The definitive benchmark for AI agents on OpenClaw. 45 tasks across 4 tiers. Powered by MyClaw.ai

rag-arenaโ˜… 222

Open-source RAG evaluation through users' feedback

AgentMeasureโ˜… 219

Inspect Codex / Claude logs, verify metering samples and re-count supported billing exports. Free local tools; bounded $10 diagnosis and $20 QA.

frugonโ˜… 218

Free, local, open-source LLM cost analyzer - see where your LLM bill leaks, on your machine.

challengeโ˜… 217

static trackio open-reproductions icml2026 agent-collab region:us

error-discovery-skillโ˜… 216

Interactive error analysis skill for AI agents. Studies LLM trace datasets, builds a review UI, monitors annotations, categorizes failure modes, proposes new samples.

SkillCompassโ˜… 214

Evaluate agent skill quality. Find the weakest link. Fix it. Prove it worked.

claudetopโ˜… 212

htop for your Claude Code sessions โ€” real-time cost, cache efficiency, model comparison, and smart alerts

agent-inspectโ˜… 212

Local evidence debugger and trajectory-test toolkit for TypeScript AI agents: inspect causal runs, catch wrong tool paths in CI, and share safe offline evidence.

lmringโ˜… 207

Open-source, self-hostable LLM arena with model compare, voting, and leaderboards

ComfyBenchโ˜… 206

Implementation for the paper "ComfyBench: Benchmarking LLM-based Agents in ComfyUI for Autonomously Designing Collaborative AI Systems".

graphsignal-pythonโ˜… 205

Graphsignal Python SDK

ai-agent-benchmark-compendiumโ˜… 203

Compendium of over 50 benchmarks for evaluating AI agents, categorized into Function Calling & Tool Use, General Assistant & Reasoning, Coding & Software Engineering, and Computer Interaction.

paramountโ˜… 202

Agent accuracy measurements for LLMs

AgentCompassโ˜… 197

[EMNLP 2026] AgentCompass is an extensible open-source evaluation infrastructure for systematically assessing LLM/VLM agent capabilities.

toktrackโ˜… 196

Ultra-fast token & cost tracker for LLM Token Usage (e.g. Claude Code)

codex-usage-trackerโ˜… 196

Local-first MCP tools and dashboard for investigating Codex token usage, credits, costs, caching, and thread patterns.

tracecrateโ˜… 195

Local-first AI agent trace workbench. Inspect Claude Code, Codex and OTLP logs, compare runs, and export privacy-conscious reports. No backend or API keys.

EvalForgeโ˜… 194

Evaluator-neutral AI evaluation evidence, baseline regression gates, and JSON, JUnit, and SARIF reports for CI.

uaefโ˜… 194

Universal Agent Evaluation Framework (UAEF) is a framework-agnostic evaluation system for AI agents. Invoke any agent and score it on multiple metrics spanning tool calling, response quality, safety, performance, and reasoning. Track experiments against baselines, catch regressions automatically, and get LLM-generated insights.

agentic-metricโ˜… 193

A local-only monitoring tool for AI coding agents.

Yet_Another_LLM_Leaderboardโ˜… 193

docker leaderboard region:us

Agent-Questโ˜… 186

Open-source fantasy village for monitoring Claude Code and Codex across terminals, apps and IDEs. Live activity, context gauges, skills, plugins and subagent families.

SkillLensโ˜… 185

SkillLens: a framework for studying model-generated agent skills across the full raw experience generation โ†’ skill extraction โ†’ skill consumption lifecycle.

vitabenchโ˜… 184

[ICLR 2026] VitaBench: Benchmarking LLM Agents with Versatile Interactive Tasks in Real-world Applications

Open-Arabic-LLM-Leaderboardโ˜… 184

gradio leaderboard region:us

UHGEvalโ˜… 183

[ACL 2024] User-friendly evaluation framework: Eval Suite & Benchmarks: UHGEval, HaluEval, HalluQA, etc.

dsh-deepcanaryโ˜… 183

Know when your DeepSeek Harness tasks need you. Local alerts, quiet notifications, and a task inbox.

ComplexFuncBenchโ˜… 182

Complex Function Calling Benchmark.

ai-coding-lang-benchโ˜… 182

Which programming language is best for AI coding agents? Benchmarking 13 languages with Claude Code.

skunitโ˜… 181

skUnit is a testing tool for AI units, such as IChatClient, MCP Servers and agents.

django-orbitโ˜… 181

AI agent-native observability and debugging for Django.

codex-pacerโ˜… 180

A Dynamic Island for Codex, Claude Code, DeepSeek Harness and WorkBuddy, on Mac and Windows. See every running task, its progress and your quota without leaving what you're doing.

quota-axiโ˜… 179

Allow your agents to see your LLM subscription quota windows.

TraceLabโ˜… 178

An open toolkit and public dataset hub for collecting, sanitizing, analyzing, and visualizing coding agent traces.

Hegelionโ˜… 176

Dialectical reasoning architecture for LLMs (Thesis โ†’ Antithesis โ†’ Synthesis)

BEAMโ˜… 176

[ICLR 2026] Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs

voice-labโ˜… 175

Testing and evaluation framework for voice agents

LLM-Agent-Benchmark-Listโ˜… 175

A banchmark list for evaluation of large language models.

langsmith-docsโ˜… 174

This repo is deprecated. Please go to langchain-ai/docs.

More in Agent frameworks & infrastructure