Agent frameworks & infrastructure
Observability & evaluation
5,667 agents ยท most starred first
Observability & evaluation, page 3 of 57
Open source AI Agent evaluation framework for web tasks ๐๐
RL environments + evals for AI agents. Define once, train anything.
Metrics to evaluate the quality of responses of your Retrieval Augmented Generation (RAG) applications.
(ACL 2025 Main) Code for MultiAgentBench : Evaluating the Collaboration and Competition of LLM agents https://www.arxiv.org/pdf/2503.01935
Instill trust in your agent evals.
[NeurIPS 2023 Datasets and Benchmarks Track] LAMM: Multi-Modal Large Language Models and Applications as AI Agents
๐ Improve your agent harness autonomously overnight
Track how your AI agents work, what they cost, and your agents heatmaps. Stop guessing your AI spend.
ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code (https://arxiv.org/abs/2311.09835)
[ACL2024] T-Eval: Evaluating Tool Utilization Capability of Large Language Models Step by Step
A General-Purpose Environment Framework for Scalable Multimodal Agent Evaluation and RL
Agent trace and tool-use safety evaluation lab.
Harness for running and evaluating AI agents against RL environments
Open-source observability dashboard for OpenClaw daemons โ cost analyti cs, live monitor ing, and de bugging tools, all running on your own machine.
Record and Replay AI Agent Trajectories.
A multi-player tournament benchmark that tests LLMs in social reasoning, strategy, and deception. Players engage in public and private conversations, form alliances, and vote to eliminate each other
X-ray vision for AI agent sessions โ a transparent HTTP proxy and dashboard for Claude Code
Build, evaluate and observe LLM apps
Measuring agents' ability to get work done on a computer
ToolQA, a new dataset to evaluate the capabilities of LLMs in answering challenging questions with external tools. It offers two levels (easy/hard) across eight real-life scenarios.
top-style TUI for monitoring AI coding agent sessions
ReactBench is an evaluation for coding agents on realistic React work
Benchmarking library for RAG
Realistic examples of building evals and optimizing agents with Harbor
AI lifecycle platform where engineers and agents track experiments, train models, and ship to production.
static region:us
Multi-Faceted AI Agent and Workflow Autotuning. Automatically optimizes LangChain, LangGraph, DSPy programs for better quality, lower execution latency, and lower execution cost. Also has a simple agent/workflow framework
Towards Large Multimodal Models as Visual Foundation Agents
Benchmarking Agentic LLM and VLM Reasoning On Games
GPU Profiler for AI agents
A repo built for the purpose of benchmarking the performance of agents, regardless of how they are set up and how they work.
Mission Control is a simple, real-time web UI for monitoring agents and task workflows. Built with Convex and React, it provides a clean dashboard to track task state, agent activity, and live logs โ all synchronized instantly without polling or message queues.
AI Initiative Ledger โ local spend meter and budget gate for AI tasks. Meters every vendor leg (LLM, speech, telephony, search, tools), hard-stops before the ceiling. MIT, no account.
[Public preview] Externally scored agentic ML research benchmark: 60 tasks, real competition ground truth. Open protocol, operated evaluation.
Open Source AI Tracing Framework built on Opentelemetry for AI Applications and Frameworks
Local-first telemetry collector for AI coding agents โ unified OpenTelemetry events for Claude Code, Codex, Cursor and more. Token usage, cost, traces and security audit, exported anywhere.
Continuous Integration for LLM powered applications
LangFair is a Python library for conducting use-case level LLM bias and fairness assessments
A comprehensive evaluation framework for AI agents and LLM applications.
Shrinkflation detector for Codex
Make AI agents and AI workflows measurably reliable. Epistemic measurement, Noetic RAG, Sentinel gating, and grounded calibration for Claude Code and beyond
Track the carbon footprint of your Claude Code sessions, in the terminal and the Claude desktop app
๐ชข Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
Spend fewer tokens on the same work. A local CLI that finds wasted tokens in Claude Code, Codex, opencode and your own API calls.
Hallucinations (Confabulations) Document-Based Benchmark for RAG. Includes human-verified questions and answers.
Multi-agent orchestrator for tracking and analyzing AI coding assistant conversations (Claude Code, Cursor, Windsurf)
Graphsignal Profiler
static leaderboard portuguese submission:closed test:public judge:auto modality:text eval:generation
A New End-to-end Framework for Evaluating Voice Agents
Benchmark CLI for comparing AI coding agents on TypeScript workloads.
:sparkles: A plotter for reinforcement learning (RL)
Find your agents errors be fore your real users do
Run your real agent with and without your skills, MCPs, and rules. See which ones actually help, and what they cost in tokens. Supports Claude Code, Codex, Pi, Hermes, Antigravity, and OpenCode.
A mini-framework for evaluating LLM performance on the Bulls and Cows number guessing game, supporting multiple LLM providers.
[ICLR 2025] A trinity of environments, tools, and benchmarks for general virtual agents
gradio leaderboard region:us
Fast, cross-platform, real-time token usage tracker and cost monitor for Claude Code / Codex CLI / Antigravity CLI / Qwen Code / Cline / Zoo Code / Kilo Code / GitHub Copilot / OpenCode / Pi Agent / Piebald.
Make AI agents and AI workflows measurably reliable. Epistemic measurement, Noetic RAG, Sentinel gating, and grounded calibration for Claude Code and beyond
Ranking LLMs on agentic tasks
ScoutAPM Ruby Agent. Supports Rails, Sinatra, Grape, Rack, and many other frameworks
The definitive benchmark for AI agents on OpenClaw. 45 tasks across 4 tiers. Powered by MyClaw.ai
Open-source RAG evaluation through users' feedback
Inspect Codex / Claude logs, verify metering samples and re-count supported billing exports. Free local tools; bounded $10 diagnosis and $20 QA.
Free, local, open-source LLM cost analyzer - see where your LLM bill leaks, on your machine.
static trackio open-reproductions icml2026 agent-collab region:us
Interactive error analysis skill for AI agents. Studies LLM trace datasets, builds a review UI, monitors annotations, categorizes failure modes, proposes new samples.
Evaluate agent skill quality. Find the weakest link. Fix it. Prove it worked.
htop for your Claude Code sessions โ real-time cost, cache efficiency, model comparison, and smart alerts
Local evidence debugger and trajectory-test toolkit for TypeScript AI agents: inspect causal runs, catch wrong tool paths in CI, and share safe offline evidence.
Open-source, self-hostable LLM arena with model compare, voting, and leaderboards
Implementation for the paper "ComfyBench: Benchmarking LLM-based Agents in ComfyUI for Autonomously Designing Collaborative AI Systems".
Graphsignal Python SDK
Compendium of over 50 benchmarks for evaluating AI agents, categorized into Function Calling & Tool Use, General Assistant & Reasoning, Coding & Software Engineering, and Computer Interaction.
Agent accuracy measurements for LLMs
[EMNLP 2026] AgentCompass is an extensible open-source evaluation infrastructure for systematically assessing LLM/VLM agent capabilities.
Ultra-fast token & cost tracker for LLM Token Usage (e.g. Claude Code)
Local-first MCP tools and dashboard for investigating Codex token usage, credits, costs, caching, and thread patterns.
Local-first AI agent trace workbench. Inspect Claude Code, Codex and OTLP logs, compare runs, and export privacy-conscious reports. No backend or API keys.
Evaluator-neutral AI evaluation evidence, baseline regression gates, and JSON, JUnit, and SARIF reports for CI.
Universal Agent Evaluation Framework (UAEF) is a framework-agnostic evaluation system for AI agents. Invoke any agent and score it on multiple metrics spanning tool calling, response quality, safety, performance, and reasoning. Track experiments against baselines, catch regressions automatically, and get LLM-generated insights.
A local-only monitoring tool for AI coding agents.
docker leaderboard region:us
Open-source fantasy village for monitoring Claude Code and Codex across terminals, apps and IDEs. Live activity, context gauges, skills, plugins and subagent families.
SkillLens: a framework for studying model-generated agent skills across the full raw experience generation โ skill extraction โ skill consumption lifecycle.
[ICLR 2026] VitaBench: Benchmarking LLM Agents with Versatile Interactive Tasks in Real-world Applications
gradio leaderboard region:us
[ACL 2024] User-friendly evaluation framework: Eval Suite & Benchmarks: UHGEval, HaluEval, HalluQA, etc.
Know when your DeepSeek Harness tasks need you. Local alerts, quiet notifications, and a task inbox.
Complex Function Calling Benchmark.
Which programming language is best for AI coding agents? Benchmarking 13 languages with Claude Code.
skUnit is a testing tool for AI units, such as IChatClient, MCP Servers and agents.
AI agent-native observability and debugging for Django.
A Dynamic Island for Codex, Claude Code, DeepSeek Harness and WorkBuddy, on Mac and Windows. See every running task, its progress and your quota without leaving what you're doing.
Allow your agents to see your LLM subscription quota windows.
An open toolkit and public dataset hub for collecting, sanitizing, analyzing, and visualizing coding agent traces.
Dialectical reasoning architecture for LLMs (Thesis โ Antithesis โ Synthesis)
[ICLR 2026] Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs
Testing and evaluation framework for voice agents
A banchmark list for evaluation of large language models.
This repo is deprecated. Please go to langchain-ai/docs.