Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 2 of 57

re_gent★ 809

Version control for AI agents — track what your agent did, blame any line to a prompt, inspect any step.

prompt-layer-library★ 790

🍰 PromptLayer - Maintain a log of your prompts and OpenAI API requests. Track, debug, and replay old completions.

auto-evaluator★ 783
onWatch★ 774

Track AI API quotas across Synthetic, Z.ai, Anthropic (Claude Code), Codex, GitHub Copilot & Antigravity in real time. Lightweight background daemon (<50MB RAM), SQLite storage, Material Design 3 dashboard. Zero telemetry.

dingo★ 771

Dingo: A Comprehensive AI Data, Model and Application Quality Evaluation Tool

agent-as-a-judge★ 768

👩‍⚖️ Agent-as-a-Judge: The Magic for Open-Endedness

overmind★ 766

The platform for continuously improving AI agents.

agentevals★ 761

Readymade evaluators for agent trajectories

agentacct★ 756

See what your coding agents did and what it cost. Breaks each task down into work steps — tools used, files changed, tests run, time and tokens spent. Local-first dashboard for Claude Code, Codex, OpenCode, and more. No login, no telemetry.

Trace★ 738

End-to-end Generative Optimization for AI Agents

agents-observe★ 738

Real-time observability of claude code sessions & multi-agents.

langchain-visualizer★ 735

Visualization and debugging tool for LangChain workflows

infinite-monitor★ 724

Monitor anything in real time

opik-openclaw★ 723

🦞 Official plugin for OpenClaw that exports agent traces to Opik. See and monitor agent behaviour, cost, tokens, errors and more.

openclaw-dashboard★ 703

🔐 Secure, real-time monitoring dashboard for OpenClaw AI agents. Auth, TOTP MFA, cost tracking, live feed, memory browser and more.

agentscope-studio★ 674

A development-oriented visualization toolkit

Awesome-LLM-Eval★ 661

Awesome-LLM-Eval: a curated list of tools, datasets/benchmark, demos, leaderboard, papers, docs and models, mainly for Evaluation on LLMs. 一个由工具、基准/数据、演示、排行榜和大模型等组成的精选列表,主要面向基础大模型评测,旨在探求生成式AI的技术边界.

Pulse★ 645

Know how much Claude Code, Codex, Cursor, Copilot and 70+ other AI coding tools you have left — a free, open-source macOS monitor on the edge of your screen.

Tracely★ 643

Trace-native CI/CD for AI agents — production failures become regression tests that block the PR. Auto-detect, cluster, freeze into hermetic cases, replay in CI for $0.

TrustLLM★ 635

[ICML 2024] TrustLLM: LLM trustworthiness evaluation across truthfulness, safety, fairness, robustness, privacy and ethics. Python/CLI toolkit for local models, compatible APIs and AI agent orchestration.

PostTrainBench★ 627

Measuring how well CLI agents like Claude Code or Codex CLI can post-train base LLMs on a single H100 GPU in 10 hours

MLGym★ 625

MLGym A New Framework and Benchmark for Advancing AI Research Agents

cc-lens★ 602

Local analytics dashboard for Claude Code. No cloud, no telemetry.

agentlytics★ 598

Comprehensive analytics dashboard for AI coding agents — Cursor, Windsurf, Claude Code, VS Code Copilot, Zed, Antigravity, OpenCode, Command Code

can-ai-code★ 597

Self-evaluating interview for AI coders

watch-skill★ 596

Give AI agents eyes, ears, and verifiable results. Watch Skill turns video, audio and screen activity into searchable, timestamped evidence and proves work with deterministic contracts, not model opinion. DeepWatch is the agent workspace built on DeepSeek Harness. Python + npm, MCP, CLI, REST, Web.

llm-perf-leaderboard★ 593

gradio llm perf leaderboard llm performance leaderboard llm performance leaderboard region:us

open-ko-llm-leaderboard★ 567

gradio leaderboard region:us

Multi-Modality-Arena★ 566

Chatbot Arena meets multi-modality! Multi-Modality Arena allows you to benchmark vision-language models side-by-side while providing images as inputs. Supports MiniGPT-4, LLaMA-Adapter V2, LLaVA, BLIP-2, and many more!

appworld★ 557

🌍 AppWorld: A Controllable World of Apps and People for Benchmarking Function Calling and Interactive Coding Agent, ACL'24 Best Resource Paper.

token-tracker★ 539

Claude Code & Codex 本地 token 追踪 — 状态栏(Codex 业界首创伪 statusline)、GitHub 风格热力图、多模型成本分析。 | Local token tracker for Claude Code & Codex — status line (industry-first Codex faux statusline), GitHub-style heatmap, multi-model cost analysis

WildClawBench★ 535

An in-the-wild benchmark for AI agents in the production harness.

TokenEater★ 529

Native macOS app to monitor AI usage limits and watch your coding sessions live

kolo★ 524

Kolo is a text-based Python debugger for AI agents. Capture every executed function call, return value, local variable, HTTP request, and more in greppable trace files.

continuous-eval★ 519

Data-Driven Evaluation for LLM-Powered Applications

mcp-bench★ 517

MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers

LLM-Performance-Leaderboard★ 504

static region:us

SWE-bench_Pro-os★ 496

SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?

skillgrade★ 493

"Unit tests" for your agent skills

hermes-agent-mission-control★ 491

Hermy HQ — a self-hostable mission-control dashboard template that pairs with your own local Hermes agent over a Postgres message bus. Send it to your Hermes and it onboards you step by step.

kibitz★ 490

Real-time decoded feed of AI agent actions — monitor multiple Claude Code & Codex sessions, see exactly what each agent is doing, and coordinate swarms efficiently

Toolathlon★ 490

[ICLR 2026] The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution

visualwebarena★ 489

VisualWebArena is a benchmark for multimodal agents.

gauntlet-loop★ 486

Turn any goal into a short prompt that makes your agent set a real quality bar, run builder and critic pairs, compare blind, and loop until it wins.

mcpmark★ 475

MCPMark is a comprehensive, stress-testing MCP benchmark designed to evaluate model and agent capabilities in real-world MCP use.

AutoHarness★ 470

AutoHarness: Automated Harness Engineering for AI Agents

rsi_eval★ 470

A lightweight Python SDK and CLI for evaluating recursive self-improvement (RSI) methods across agent benchmarks.

ParseBench★ 464

ParseBench - A Document Parsing Benchmark for AI Agents

openclaw-dashboard★ 463

A beautiful, zero-dependency command center for OpenClaw AI agents

phasellm★ 455

Large language model evaluation and workflow framework from Phase AI.

AgentBoard★ 454

An Analytical Evaluation Board of Multi-turn LLM Agents [NeurIPS 2024 Oral]

agent-leaderboard★ 454

gradio region:us

claude-code-cache-fix★ 451

Fixes prompt cache regression in Claude Code that causes up to 20x cost increase on resumed sessions

AgentEval★ 449

The agent responsible for conducting the agent evaluation

agentenv-framework★ 447

Creating realistic RL environments requires collaboration between researchers, engineers, and domain experts across many dimensions: artifacts, environments tools, dynamism of the environment, reproducibility, and more. There is no open source framework for building these environments effectively. Until now.

frontier-bench★ 442

Measuring and evolving with the frontier of agent work

MAST★ 440
clawmetry★ 439

See your agent think. Zero-config observability & governance for 30 AI agent runtimes: Claude Code, OpenAI Codex, Hermes, OpenClaw & 26 more. Live token costs, sessions, tool calls, crons.

open_medical_llm_leaderboard★ 437

gradio region:us

crab★ 435

🦀️ CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents. https://crab.camel-ai.org/

tool-eval-bench★ 435

Tool-calling quality benchmark for LLM serving stacks. 80+ deterministic scenarios testing multi-turn orchestration, safety boundaries, and structured output. Supports vLLM, SGLang, and llama.cpp.

spring-ai-alibaba-admin★ 427

Console supporting tracing, prompt engineering and evaluation for Agents developed with Spring AI Alibaba.

open-rag-eval★ 427

RAG evaluation without the need for "golden answers"

RAMPART★ 427

A pytest-native safety and security testing framework for agentic AI applications

perfmon-agent★ 419

Server metrics fetching agent, based on SIGAR

pykoi★ 410

pykoi: Active learning in one unified interface

agent-prism★ 408

React components for visualizing traces from AI agents

whatsmyagentdoing★ 408

See what every Claude Code and Codex agent on your machine is doing — live, in a little office.

rhesis★ 405

The collaboration layer for teams building AI agents. Test, simulate, and trace your agents, while domain experts annotate behavior and engineers improve the agent from what they find.

AutomationBench★ 404

A benchmark for evaluating AI agents on realistic business workflows

tiger★ 403

Open Source LLM toolkit to build trustworthy LLM applications. TigerArmor (AI safety), TigerRAG (embedding, RAG), TigerTune (fine-tuning)

pi-harness★ 399

The operational superset of Pi Coding Agent — everything Pi, plus observability, governance, recovery, evaluation and multi-agent orchestration. Pi Coding Agent 的运维超集——完整继承 Pi 的核心能力,并进一步扩展可观测性、治理、故障恢复、评估和多 Agent 编排。

Claude-Office★ 397

A pixel art virtual office that visualizes your AI agents working in real-time. Claude Code hooks, WebSocket events, and isometric pixel art.

Agents_Failure_Attribution★ 396

Benchmark for automated failure attributions in agentic systems (🏆 ICML 2025 Spotlight)

herdr-mobile-relay★ 390

Approve and monitor Herdr agents remotely from your phone, a mobile web app for Android/iOS smartphones with push notifications, QR setup, and multi-computer relays over Cloudflare, WebRTC or Tailscale connections.

tokentelemetry★ 387

Token telemetry dashboard for AI autonomous and coding agents — tracks tokens, sessions, tool calls & reasoning across Hermes agent, Claude Code, Antigravity CLI, Codex & more. 100% local.

gorelic★ 383

GoRelic is deprecated in favour of https://github.com/newrelic/go-agent

awesome-ai-leaderboard★ 383

A curated list of awesome leaderboard-oriented resources for AI domain

agentbehavior★ 379

Standards for defining and evaluating agent behavior

agent-evaluation★ 377

A generative AI-powered framework for testing virtual agents.

agentplay★ 372

Replay your coding agent sessions as a map and a timeline

AdaRubrics★ 369

AdaRubric: Adaptive Dynamic Rubric Evaluator for Agent Trajectories

ASSERT★ 367

Requirement-driven evaluation harness for AI agents and LLM applications. Generate behavior-specific test cases, run them against any target (hosted models, callable wrappers, OTel-traced agents), and inspect local-first artifacts.

TokenBar★ 365

AI token usage & quota monitor for the macOS menu bar — native Swift, Liquid Glass, 3D contribution graph. Tracks Claude Code, Codex, Cursor, OpenCode & 25+ agents locally.

multi-swe-bench★ 364

Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving

skywalking-go★ 362

The Golang auto-instrument Agent for Apache SkyWalking, which provides the native tracing/metrics/logging abilities for Golang projects.

MLAgentBench★ 360
EmbodiedBench★ 360

[ICML 2025 Oral] Official repo of EmbodiedBench, a comprehensive benchmark designed to evaluate MLLMs as embodied agents.

llm-leaderboard★ 356

A comprehensive set of LLM benchmark scores and provider prices. (deprecated, read more in README)

usage★ 352

macOS menu bar & Windows tray app pinning Claude Code, Codex, Antigravity & Grok CLI quota, burn rate, and cost to your screen. Local-first, no telemetry. HTML reports, 16 themes.

gpu-poor-llm-arena★ 349

gradio region:us

agentglass★ 349

🛰 Every AI coding agent on your machine, on one screen — live cost, tokens and tool calls across every provider, and a hold on anything dangerous until you say go. From your desk or your phone.

agentsight★ 348

Zero instrucment LLM and AI agent (e.g. claude code, openclaw, gemini-cli) observability in eBPF

syftr★ 346

syftr is an agent optimizer that helps you find the best agentic workflows for your budget.

ChatEval★ 346

Codes for our paper "ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate"

sotopia★ 346

Sotopia: an Open-ended Social Learning Environment (ICLR 2024 spotlight)

monocle★ 345

Monocle is a framework for tracing GenAI app code. This repo contains implementation of Monocle for GenAI apps written in Python.

palico-ai★ 343

Build, Improve Performance, and Productionize your AI Application

skills★ 343

Agent Skills for Langfuse, the open source LLM engineering platform for tracing, prompt management, and evaluation

openlogs★ 339

give agents access to your logs

More in Agent frameworks & infrastructure