Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 4 of 57

AgentGuard★ 174

Real-time guardrail that shows token spend & kills runaway LLM/agent loops.

ClaudeMeter★ 174

macOS menu bar app for tracking Claude.ai plan usage in real-time. Monitor 5-hour session, 7-day weekly, and Sonnet-specific limits with color-coded indicators and smart notifications.

toktop★ 173

llm usage monitor in terminal

benchmark★ 173

Open benchmarks for browser agents: BU Bench V2.1, 200 real web tasks. Rerun them with your own API keys.

rageval★ 172

Evaluation tools for Retrieval-augmented Generation (RAG) methods.

agentevals★ 171

agentevals is a framework-agnostic evaluations solution based on OpenTelemetry traces

Kaggle-4th-Place-Solution-LMSYS-Chatbot-Arena-Human-Preference-Predictions★ 169

4th Place Solution for the Kaggle Competition: LMSYS - Chatbot Arena Human Preference Predictions

agent-memory-benchmark★ 169

Agent Memory Benchmark

One-Eval★ 165

Automated system for LLM evaluation via agents. Doc as below:

NiceEval★ 165

build eval for your agent in 10 mins

evalkit★ 164

The TypeScript LLM Evaluation Library

NewtonBench★ 164

[ICLR2026] NewtonBench: Benchmarking Generalizable Scientific Law Discovery in LLM Agents

ZizkaDB★ 164

Audit trail database for AI agents. Tamper-evident, checksum-backed decision logs with session replay and time-travel debugging to support EU AI Act Article 12 record-keeping. Drift detection, MCP, Python & TypeScript SDKs. Self-host or cloud.

AgentEval★ 163

AgentEval is the comprehensive .NET toolkit for AI agent evaluation—tool usage validation, RAG quality metrics, stochastic evaluation, and model comparison—built first for Microsoft Agent Framework (MAF) and Microsoft.Extensions.AI. What RAGAS, PromptFoo and DeepEval do for Python, AgentEval does for .NET

SAG-Benchmark★ 162

Official benchmark reproduction code for the SAG paper

awesome-auditable-ai★ 162

Auditing AI agents: a curated list of papers, tools, datasets, benchmarks, and standards covering reliability, monitoring, failure attribution, and decision records.

herdr-agent-usage★ 162

Credential-scoped AI usage, context, and cache in Herdr for Claude, Codex, Grok, Agy, OpenCode, Pi, omp, Devin, Muse, MiMo, Kilo, Antigravity, and Cursor.

WorfBench★ 161

[ICLR 2025] Benchmarking Agentic Workflow Generation

RouterArena★ 159

RouterArena: An open framework for evaluating LLM routers with standardized datasets, metrics, an automated framework, and a live leaderboard.

OpenTracy★ 158

Self-improving AI agent harness — propose, eval, approve, ship. AHE autonomous loop + MCP + BYOK + eval suite.

agenttrace★ 157

Local-first Rust TUI/CLI for auditing AI coding-agent sessions: cost, tokens, latency, failures, and health|本地运行的 Rust TUI/CLI,审计 AI 编程 Agent 会话的成本、Token、延迟、失败与健康度

token-meter★ 154

Open-source, local-first AI coding agent usage and cost dashboard for Claude Code, Codex, Cursor, OpenCode, Kiro, and Pi.

GTA★ 153

[NeurIPS 2024 D&B] GTA: A Benchmark for General Tool Agents & [arXiv 2026] GTA-2

BixBench★ 153

Benchmark for LLM-based Agents in Computational Biology

EvoPolicyGym★ 153

EvoPolicyGym is infrastructure for evaluating coding agents and generating training experience through Autonomous Policy Evolution.

superpowers-evals★ 150

Behavioral eval lab (Quorum) for the superpowers project that drives real coding-agent CLIs (Claude, Codex, Gemini, Kimi, and more) through a QA agent and grades them on workflow compliance against scenario criteria and deterministic post-checks.

nemo-helix★ 150

NeMo Helix: Make the agents you ship faster, more accurate, and safer.

Agent-Eval-Refine★ 149

Code for Paper: Autonomous Evaluation and Refinement of Digital Agents [COLM 2024]

llm-speedrunner★ 148

The Automated LLM Speedrunning Benchmark measures how well LLM agents can reproduce previous innovations and discover new ones in language modeling.

SmartPlay★ 145

SmartPlay is a benchmark for Large Language Models (LLMs). Uses a variety of games to test various important LLM capabilities as agents. SmartPlay is designed to be easy to use, and to support future development of LLMs.

ChatOpsLLM★ 143

To simplify and streamline LLM operations, empowering developers and organizations to harness the full potential of large language models with ease.

best-llm-finder-pipeline★ 143

Agentic RAG, Multi-Agent Systems, and Vision Reasoning are three pipelines to find the perfect LLM

eval-view★ 142

Regression testing for AI agents. Snapshot behavior,diff tool calls,catch regressions in CI. Works with LangGraph, CrewAI, OpenAI, Anthropic.

eval-guide★ 142

A plugin for AI agent evaluation. Plan evals, generate test cases, interpret results for Copilot Studio agents. Grounded in Microsoft's Eval Scenario Library & Triage Playbook.

contextrot★ 138

Find out where your coding agent starts degrading. Personal context-rot analytics from your own sessions - 100% local, zero-config: uvx contextrot

infomarchy★ 138

Omarchy plugin: your wallpaper becomes a live, clickable information desk with every running AI agent (Claude Code, Codex, Grok, Ollama), 7-day heatmap, rate limits, recent prompts, machine stats. Theme-driven.

c9watch★ 137

Mac app and JSON CLI to monitor every Claude Code, Codex, Cursor Agent and Pi session on your machine. See which agent needs permission, which is working, and which is idle. Works with any terminal or IDE. Free, open source (MIT).

openbench★ 137

Same model, different wrapper: a from-scratch benchmark comparing coding-agent harnesses (codex, pi, opencode, cursor, devin) and open models on correctness, speed, and token cost

CUREBench★ 136

CUREBench @ NeurIPS 2025: Benchmarking AI reasoning for therapeutic decision-making at scale

copilot-arewecooked★ 136

Estimate your GitHub Copilot AI-credit cost in preparation for June 1st.

opsrobot★ 135

Observability platform for Digital Employee, providing real-time tracing, session insights, and cost analysis for multi-agent workflows

steer★ 135

The Active Reliability Layer for AI Agents. Catch failures, teach fixes, and automate reliability

traccia-py★ 134

OpenTelemetry-native SDK for AI agent observability, tracing, evaluation, debugging, governance, and runtime policy enforcement. Framework-agnostic and built for OpenAI Agents, LangGraph, CrewAI, and LLM applications in production.

aa-agentperf-local★ 134

Benchmark local LLM serving by replaying real agent trajectories

RAGElo★ 133

RAGElo is a set of tools that helps you selecting the best RAG-based LLM agents by using an Elo ranker

MuseAi-Skills-Guide★ 130

(THIS IS THE PUBLIC VERSION) Muse-by-Meta usage audit companion designed for those who want to understand Muse’s actions during a multi-step task without pretending to have telemetry they can’t actually access.Check out Muse, your personal AI agent. Code: 5F2V31 https://muse.ai/join

Info-Mosaic★ 129

[ICLR 2026] InfoMosaic-Bench: Evaluating Multi-Source Information Seeking in Tool-Augmented Agents

tool-calling-benchmark★ 129
agent-learning-kit★ 129

General Purpose Evaluation and Simulation Environment for all your AI related Workflows

OPT-BENCH★ 128

[ACL 2026] OPT-BENCH: Evaluating the Iterative Self-Optimization of LLM Agents in Large-Scale Search Spaces

ClawMark★ 128

🦞 ClawMark: A Living-World Benchmark for Multi-Day, Multimodal Coworker Agents

claude-view★ 128

10 Claude sessions running. What are they doing? Live dashboard — monitor, cost tracking, search, sub-agent visibility.

open_cn_llm_leaderboard★ 128

gradio region:us

agentwatch★ 127

A powerful AI observability framework that provides comprehensive insights into agent interactions across platforms, enabling developers to monitor, analyze, and optimize AI-driven applications with minimal integration effort.

airs-bench★ 127

AIRS-Bench: an AI Research Science benchmark for quantifying the end-to-end AI research abilities of LLM agents

agentcat-typescript-sdk★ 127

AgentCat is an analytics platform for MCP server owners 🐱.

berkeley-function-calling-leaderboard★ 127

static leaderboard eval:code modality:tools modality:text region:us

otelyssey★ 127

OpenTelemetry plugins for every coding agent, in the Agent Plugins format — a marketplace that runs itself: open an issue, the repository validates, reviews, lists and follows your plugin.

awesome-ai-eval★ 127

Assessing the reliability, accuracy, and safety of large language models, RAG pipelines, and AI agents.

cc-bar★ 126

macOS 原生菜单栏 AI 额度监控与本地用量看板 实时追踪 Codex、Claude Code、Antigravity、Cursor 与 Command Code 配额状态,精准分析本地会话 Token 与费用。

tma1★ 125

Local-first observability your agent reads back. TMA1 records every LLM call, then routes what it sees into the agent's next turn via hooks and MCP.

tokenjam★ 125

Token Efficiency For AI Agents

observatory★ 124
error-discovery-skill★ 123

Interactive error analysis skill for AI agents. Studies LLM trace datasets, builds a review UI, monitors annotations, categorizes failure modes, proposes new samples.

toiro★ 122

A tool for comparing tokenizers

primitive-bench★ 122

The marketplace for verifiable AI outcomes: state a task, get a fixed price upfront, and Primitive Bench routes across tools to complete it, refunded if the outcome isn't delivered. This repo is the open verification layer.

agent-trust★ 121

🤝 The code for "Can Large Language Model Agents Simulate Human Trust Behaviors?"

viberank★ 121

🏆 The AI coding usage leaderboard — Claude Code, Codex, Gemini CLI & more. Real costs and tokens from ccusage data. Submit with: npx viberank-cli

thinkingbox★ 121

thinkingbox is a framework for defining tool as MCP servers, running LLM agents against them, and evaluating agent behavior — for offline training-data generation, reinforcement-learning training loops, and model evaluation.

codex-trace★ 120

OpenAI Codex CLI session log viewer for JSONL files in ~/.codex/sessions. Browse conversations, tool calls, tokens, collaboration chains, and live sessions on desktop and web.

agenticlens★ 120

An open-source profiler for AI agents that analyzes token usage, cost, latency, and optimization opportunities across LLM workflows.

argus★ 119

Claude Code Agent Monitoring & Observability on VSCode

AlgoTune★ 119

AlgoTune is a NeurIPS 2025 benchmark made up of 154 math, physics, and computer science problems. The goal is write code that solves each problem, and is faster than existing implementations.

AgentKernelArena★ 119

AgentKernelArena provides an end-to-end siloed-benchmarking environment where different LLM-powered agents—such as Cursor Agent, Claude Code, Codex, SWE-agent, and GEAK—can be evaluated side-by-side on the same GPU kernel tasks, using objective and reproducible metrics.

Co-Design-Bench★ 118

Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models

cronalytics★ 117

Hermes Agent plugin for cron analytics and observability. The dashboard for agentic automations in Hermes.

brier★ 117

Anyone can predict the future. Almost nobody keeps the receipt. A hash-chained prediction ledger for LLMs — sealed before the answer exists, graded with proper scoring rules.

doc2harness★ 117

Official Code for the paper "Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents "

MIRAI★ 116

Code and Data for "MIRAI: Evaluating LLM Agents for Event Forecasting"

moraine★ 116

Unified realtime agent trace database & search MCP

menu-bar-token-economy★ 116

Best AI API Cost Tracker & Token Usage Monitor for Developers 2026

token-efficiency-lex★ 116

Token Cost Parity: Multilingual LLM Efficiency Analysis 2026

fde-gym★ 116

The online judge for the LLM era. Every case is a real customer delivery: you work it with a coding agent, and your system is judged once it is live.

hallucination-index★ 115

Initiative to evaluate and rank the most popular LLMs across common task types based on their propensity to hallucinate.

clawd-control★ 115

🏰 Real-time dashboard for monitoring and managing Clawdbot AI agents

MMBench-GUI★ 115

Official repo of "MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents". It can be used to evaluate a GUI agent with a hierarchical manner across multiple platforms, including Windows, Linux, macOS, iOS, Android and Web.

litefuse★ 115

Litefuse - Agent Observability and Evaluation Platform

comparator★ 115

gradio leaderboard region:us

LLM-Agent-Evaluation-Survey★ 114

Top papers related to LLM-based agent evaluation

kieker★ 113

Kieker is an observability framework, that consists of an monitoring and tracing agent, and compatible analysis tools

hermes-webui★ 113

Process monitoring and configuration dashboard for Hermes Agent

inferock-bench★ 111

Local LLM cost-tracking proxy for OpenAI, Anthropic, Gemini, and pinned OpenRouter calls with token usage, failure, and billing-integrity receipts.

claw-swe-bench★ 111

Unified adapter framework for evaluating agent harnesses (claws) on SWE-bench

silicon-rider-bench★ 110

The KCORES Agent benchmarking project is designed to evaluate the tool-call capabilities of single-modal/multi-modal models.

sample-agent-cost-bench★ 110

Benchmark framework that measures cost, quality, and duration of coding agents across any AI Coding Assistant CLI, any model, and any use case — with pluggable verification and real-repo support.

metrik★ 110

Metrik 可以集中查看本机多个 Agent 的配额余量和 Token 消耗,目前支持 ChatGPT、Claude、GLM、Kimi 等主流 AI 服务。

NeMo-Agent-Toolkit-UI★ 109

The NVIDIA NeMo Agent Toolkit UI streamlines interacting with NeMo Agent Toolkit workflows in an easy-to-use web application.

RewardHarness★ 109

Self-evolving agentic reward framework for image-editing evaluation — 47.4% on EditReward-Bench from only 100 preference demos, no reward-model training. arXiv 2605.08703.

ReasoningEffortLLM★ 109

What Effort Level Should I Choose? On the Economics of Reasoning Effort in LLM Agents

autoarena★ 108

Rank LLMs, RAG systems, and prompts using automated head-to-head evaluation

More in Agent frameworks & infrastructure