Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 5 of 57

PCA-EVAL★ 108

[ACL 2024] PCA-Bench: Evaluating Multimodal Large Language Models in Perception-Cognition-Action Chain

agent-trace★ 108

Local evidence and replay for coding agents: replay what happened, see what the trace could not prove, and attach safe evidence to review. Claude Code, Codex, Copilot CLI, Gemini CLI, Cursor, MCP.

neatlogs★ 107

AI agent debugging, collaboration, and trace observability. Built for teams using CrewAI, OpenAI, and more.

VibeSearchBench★ 107

🔍 The hardest search benchmark in the wild — vague, multi-turn, proactive. 200 long-horizon tasks with persona-driven progressive disclosure, scored by verifiable schema-free knowledge-graph evaluation. No vibes, just triplet F1.

open-japanese-llm-leaderboard★ 107

gradio 日本語 Japanese leaderboard language:日本語 language:Japanese region:us

european-llm-leaderboard★ 107

gradio region:us

open_llm_leaderboard★ 106

gradio leaderboard region:us

tokpet★ 106

Your AI usage, quota, and balance — as a desk pet. 🐾 A local companion service that normalizes every AI provider into one LAN feed, plus ESP32-S3 firmware for the device that shows it.

benchmark-qed★ 105

Automated benchmarking of Retrieval-Augmented Generation (RAG) systems

AgencyBench★ 105

[ACL2026 Main] AgencyBench: Benchmarking the Frontiers of Autonomous Agents in 1M-Token Real-World Contexts

ccboard★ 105

Monitor Claude Code sessions, costs, config, hooks, agents & MCP servers from a single Rust binary — TUI (9 tabs) + Web interface with live process tracking, budget alerts, and 30-day forecasting

codex-reset-watcher★ 105

Local-first macOS menu bar app for Codex usage limits and reset credits.

Beagle★ 105

:dog: Beagle is designed to be the unified platform for the broader Recursive Self-Improvement (RSI) ecosystem. Think of it as Hugging Face Trainer for agent evolution. Its efficient rollout engine enables evaluation and evolution at scale across different agents, datasets, benchmarks, and evolution methods.

Arena-Baselines★ 104

Arena: A General Evaluation Platform and Building Toolkit for Single/Multi-Agent Intelligence. AAAI 2020.

self-reflection★ 104

Self-Reflection in LLM Agents: Effects on Problem-Solving Performance

heron★ 104

Agent and LLM API performance monitoring via network packet probe. Measures performance of OpenClaw, Claude, Codex, DeepAgents and more — deployed on the provider side, no SDK changes required.

kernelbench.com★ 104

kernelbench.com — GPU kernel engineering benchmarks for autonomous LLM coding agents. v3 archive + v-hard latest.

awesome-agent-trajectory★ 104

A collection of agent trajectory analysis techniques and benchmark

agent-orchestration-standard★ 104

A live, local-first dashboard for Claude Code sessions: project graph of agents, tokens, tool feed, replay, alerts. Plugin + terminal UI + MCP server. Includes the AOS-1 agent orchestration spec.

ResolveLab★ 104

Fault-injected tool environments with side-effect grading for evaluating and post-training LLM agents

ThinkingAgent★ 103

Systematic evaluation framework that automatically rates overthinking behavior in large language models.

agentsre-langchain★ 103
reforge-ai-evals★ 102

Open-source AI evals toolkit: skills that guide AI coding agents to write eval rubrics, eval datasets (User Input Grid), trace codes, code evaluators and LLM-as-a-judge prompts. From the Reforge AI Evals course by Calibre.

llm-eval★ 101

大语言模型评估平台,支持多种评估基准、自定义数据集和性能测试。支持基于自定义数据集的RAG评估。

repo-asynctool★ 101

「EMNLP2026 Findings」AsyncTool is a benchmark for evaluating the asynchronous function-calling capability of LLM-based agents in multi-task tool-use scenarios.

mcpcat-typescript-sdk★ 100

MCPcat is an analytics platform for MCP server owners 🐱.

ai-evaluation★ 100

Evaluation Framework for all your AI related Workflows

WebBench★ 100

📚 Benchmark your browser agent on ~2.5k READ and ACTION based tasks

dot-agent-deck★ 100

A rich terminal dashboard for monitoring and controlling multiple AI coding agent sessions

Kyoko★ 100

🔨 Kyoko is the all-in-one, fully local tool for debugging and improving your AI agents.

opentraces★ 100

A local-first evidence layer for agent work, capturing what the agent sees, does, and changes.

aec-bench★ 100

AEC Bench: A Multimodal Benchmark for Agentic Systems in Architecture, Engineering, and Construction

STATE-Bench★ 100

Benchmark AI Agents on Enterprise Workflows

FeatureBench★ 100

[ICLR 2026🔥] Official Implementation of "FeatureBench: Benchmarking Agentic Coding for Complex Feature Development"

egma★ 100

the first open-source platform for simulation testing, monitoring and self-improving voice agents

KUMA-DefuzeX★ 100

Python SDK for DefuzeX agent behavior testing.

lemans★ 100

A Ruby harness for running agent benchmarks

dsh-harness-evolution★ 100

Evolved DSH plugin harnesses for five agent benchmarks, with versioned artifacts, evolution curves and Life-Harness comparisons

ScoreFlow★ 99

ScoreFlow: Mastering LLM Agent Workflows via Score-based Preference Optimization

LLF-Bench★ 99

A benchmark for evaluating learning agents based on just language feedback

contextcheck★ 99

MIT-licensed Framework for LLMs, RAGs, Chatbots testing. Configurable via YAML and integrable into CI pipelines for automated testing.

AgenTracer★ 99

AgenTracer: A Lightweight Failure Attributor for Agentic Systems

CoBRA★ 99

[🏆 CHI26 Best Paper] CoBRA: Reproducible control of LLM agent behavior via classic social science experiments

LifelongAgentBench★ 99

Code repo for "LifelongAgentBench: Evaluating LLM Agents as Lifelong Learners"

ai-agent-evals★ 99

Github action to evaluate AI agent applications using model as the judge, content safety and mathematical metrics.

agent-readiness★ 99

The open-source alternative to Factory.ai's Agent Readiness.

agentic-vbench★ 99

AgenticVBench: Can AI Agents Complete Real-World Post-Production Tasks?

DataAgentBench★ 99

Benchmark for data agents

NatureBench★ 99

NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?

agento11y★ 99

Actually Useful Agent Observability

cartwheel-homeworks★ 99

Homework assignments for Evaluating and Improving AI Agents

clawd-usage-winwidget★ 99

Claude usage limits in a tiny Windows widget + tray icon — with pixel Clawd who reacts to your Claude Code sessions, looks after your PC and chats with you, from the widget or your phone. PowerShell, no dependencies.

limen★ 99

Local, opt-in proxy for coding agents: measures what your agent re-sends, compiles the scaffolding out of the prompt, and enforces nothing until you say so.

GAMABench★ 98

Code and data for the paper: Competing Large Language Models in Multi-Agent Gaming Environments

MemoryData★ 98

A Unified Memory Benchmark Suite for Memory-Augmented Agents

pi-infobar★ 98

Native macOS (menu bar) & Windows (system tray) dashboard for Pi agent usage, cost, languages, models, projects & tokens. Local-first, optional SSH sync.

dsh-watcher★ 98

Read-only Agent work-path observer for DeepSeek Harness

rsi-benchmark★ 98

RSI Bench is an ongoing effort to evaluate whether AI agents can develop the capabilities required to advance AI R&D.

burn★ 98

Local-first, zero-dependency cost ledger for your AI coding agents. Reads the transcripts Claude Code / OpenCode / Codex / Gemini CLI already leave on disk — spend per repo, model, day, and git branch, with honest billed/estimated/unpriced provenance. No network, no account, no telemetry. ULP 1.0 reference implementation.

dialog-eval★ 97

Evaluate your dialog model with 17 metrics! (see paper)

log10★ 97

Python client library for improving your LLM app accuracy

claw-swe-bench★ 97

Unified adapter framework for evaluating agent harnesses (claws) on SWE-bench

MemoryBench★ 97

Code for MemoryBench: A Benchmark for Memory and Continual Learning in LLM Systems

isplay★ 97

Understand your agent's decisions

hermes-pixel-office★ 97

Pixel-art virtual office for Hermes Agent — watch your agents work as animated pixel characters

jevals★ 97

Agent evals and guardrails as Jev decisions: one request per trace, a fraction of a cent, fast enough for the agent loop. Runs locally with Kev or Laya.

SuperCLUE-Agent★ 96

SuperCLUE-Agent: 基于中文原生任务的Agent智能体核心能力测评基准

heurigym★ 96

Agentic Benchmark for LLM-Crafted Heuristics in Combinatorial Optimization (ICLR'26)

agentopt★ 96

AgentOpt automatically finds the best LLM model combination for each step of your agent — optimizing for accuracy, cost, and latency.

ai-agent-session-center★ 96

Real-time dashboard for Claude Code and Codex AI coding agents: live terminals, status alerts, prompt queues, and workspace resume. Desktop app or browser, with an optional 3D view.

opencompass-llm-leaderboard★ 96

gradio leaderboard region:us

OpenCaptchaWorld★ 96

[NeurIPS 2025] The first web-based benchmark and platform to evaluate visual reasoning and interaction capabilities of MLLM powered agents through diverse and dynamic CAPTCHA puzzles.

sacp★ 96

State-Aware Collaboration Protocol: auditable work receipts for AI agents.

dotpals★ 96

See what your coding agent actually did. A desktop pal and notch for Claude Code, Codex, Cursor, Gemini CLI and any agent: test results read from the output, faked passes caught, and an MCP server any AI tool can ask. New: talk to Blu, who sees your screen, draws on it as it explains and uses your apps.

ares★ 95

Agentic Research and Evaluation Suite

Axon★ 95

OpenTelemetry-native LLM observability CLI. Point any OTEL exporter at it and watch your LLM/agent traces in real time.

Nexus_Function_Calling_Leaderboard★ 95

gradio region:us

open-persian-llm-leaderboard★ 95

gradio region:us

cxas-scrapi★ 95

A powerful Python API, CLI, and set of Agent Skills for CX Agent Studio to automate, evaluate, and scale your agents with ease.

dsh-maze★ 95

DeepSeek Harness 的执行迷宫——看 Agent 真实怎么干活:迷宫时间轴 · 数据轨道 · 确定性执行分析 · 多会话对比 | The execution maze for DSH agents: maze timeline, per-step data tracks, deterministic execution analysis, multi-session comparison. Formerly dsh-trace-compare.

trace2task★ 95

Turn successful workflows into resettable, verifiable benchmarks for AI agents.

TraceVista★ 95

Understand Your Agents. Turn agent logs into interactive maps and stories. Explore decisions, replay the work, and get answers linked to the original evidence. For coding, research, and creative workflows.

prompt-spider-ui★ 95

原生 JavaScript + Canvas 2D 的机械蜘蛛可视化 UI;可选 MCP 接入自己的 Agent,实时展示任务,支持三套布局、主题定制、JSON 回放与独立项目导出。 | Canvas spider UI with live Agent visualization via MCP.

claude-usage-widget★ 94

Claude usage widget for Windows

step_game★ 93

Multi-Agent Step Race Benchmark: Assessing LLM Collaboration and Deception Under Pressure. A multi-player “step-race” that challenges LLMs to engage in public conversation before secretly picking a move (1, 3, or 5 steps). Whenever two or more players choose the same number, all colliding players fail to advance.

arthur-engine★ 93

Make AI work for Everyone - Monitoring and governing for your AI/ML

coding_agent_usage_tracker★ 93

Single CLI to monitor LLM coding-agent usage across 16+ providers (Codex, Claude, Gemini, Cursor, Copilot, and more): remaining quota, rate limits, and credits as terminal tables or JSON/Markdown, plus an MCP server for agents.

pane★ 93

Pane tracker: all your AI plans in one Windows tray. Claude, Codex, Cursor, Copilot, Kimi, Grok and 15 more. Free, open source. The OpenUsage port for Windows.

skillfoo-v3★ 93

Bounded, evidence-driven SKILL.md evolution under frozen evaluation and safety contracts.

SkillLearnBench★ 93

[COLM'26] SkillLearnBench is the first benchmark for evaluating continual learning methods that automatically generate agent skills.

Composio-Function-Calling-Benchmark★ 92

Function Calling Benchmark & Testing

agent-console★ 92

Live view of Claude Code sessions and the ability to search them

openclaw-telemetry★ 92

Telemetry for OpenClaw - Captures tool calls, LLM usage, agent lifecycle, and message events. Outputs to JSONL file and optionally to syslog for SIEM integration.

inspect_scout★ 92

In-depth analysis of AI agent transcripts.

Tokdash★ 92

Agent Dashboard: Visualization and analytics for Sessions and Quota Usage. Track, analyze, and optimize token usage across providers with heatmaps, cost tracking, token counting and quota resets..

llm-trustworthy-leaderboard★ 92

gradio leaderboard region:us

Composio-Function-Calling-Benchmark★ 92

Function Calling Benchmark & Testing

DecisionTransformerInterpretability★ 91

Interpreting how transformers simulate agents performing RL tasks

lm-speed★ 91

Help developers optimize AI application performance through comprehensive speed testing and analysis

dynatrace-ai-agent-instrumentation-examples★ 91

Dynatrace AI Observability OTel instrumentation examples

More in Agent frameworks & infrastructure