Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 4 of 57
Real-time guardrail that shows token spend & kills runaway LLM/agent loops.
macOS menu bar app for tracking Claude.ai plan usage in real-time. Monitor 5-hour session, 7-day weekly, and Sonnet-specific limits with color-coded indicators and smart notifications.
llm usage monitor in terminal
Open benchmarks for browser agents: BU Bench V2.1, 200 real web tasks. Rerun them with your own API keys.
Evaluation tools for Retrieval-augmented Generation (RAG) methods.
agentevals is a framework-agnostic evaluations solution based on OpenTelemetry traces
4th Place Solution for the Kaggle Competition: LMSYS - Chatbot Arena Human Preference Predictions
Agent Memory Benchmark
Automated system for LLM evaluation via agents. Doc as below:
build eval for your agent in 10 mins
The TypeScript LLM Evaluation Library
[ICLR2026] NewtonBench: Benchmarking Generalizable Scientific Law Discovery in LLM Agents
Audit trail database for AI agents. Tamper-evident, checksum-backed decision logs with session replay and time-travel debugging to support EU AI Act Article 12 record-keeping. Drift detection, MCP, Python & TypeScript SDKs. Self-host or cloud.
AgentEval is the comprehensive .NET toolkit for AI agent evaluation—tool usage validation, RAG quality metrics, stochastic evaluation, and model comparison—built first for Microsoft Agent Framework (MAF) and Microsoft.Extensions.AI. What RAGAS, PromptFoo and DeepEval do for Python, AgentEval does for .NET
Official benchmark reproduction code for the SAG paper
Auditing AI agents: a curated list of papers, tools, datasets, benchmarks, and standards covering reliability, monitoring, failure attribution, and decision records.
Credential-scoped AI usage, context, and cache in Herdr for Claude, Codex, Grok, Agy, OpenCode, Pi, omp, Devin, Muse, MiMo, Kilo, Antigravity, and Cursor.
[ICLR 2025] Benchmarking Agentic Workflow Generation
RouterArena: An open framework for evaluating LLM routers with standardized datasets, metrics, an automated framework, and a live leaderboard.
Self-improving AI agent harness — propose, eval, approve, ship. AHE autonomous loop + MCP + BYOK + eval suite.
Local-first Rust TUI/CLI for auditing AI coding-agent sessions: cost, tokens, latency, failures, and health|本地运行的 Rust TUI/CLI,审计 AI 编程 Agent 会话的成本、Token、延迟、失败与健康度
Open-source, local-first AI coding agent usage and cost dashboard for Claude Code, Codex, Cursor, OpenCode, Kiro, and Pi.
[NeurIPS 2024 D&B] GTA: A Benchmark for General Tool Agents & [arXiv 2026] GTA-2
Benchmark for LLM-based Agents in Computational Biology
EvoPolicyGym is infrastructure for evaluating coding agents and generating training experience through Autonomous Policy Evolution.
Behavioral eval lab (Quorum) for the superpowers project that drives real coding-agent CLIs (Claude, Codex, Gemini, Kimi, and more) through a QA agent and grades them on workflow compliance against scenario criteria and deterministic post-checks.
NeMo Helix: Make the agents you ship faster, more accurate, and safer.
Code for Paper: Autonomous Evaluation and Refinement of Digital Agents [COLM 2024]
The Automated LLM Speedrunning Benchmark measures how well LLM agents can reproduce previous innovations and discover new ones in language modeling.
SmartPlay is a benchmark for Large Language Models (LLMs). Uses a variety of games to test various important LLM capabilities as agents. SmartPlay is designed to be easy to use, and to support future development of LLMs.
To simplify and streamline LLM operations, empowering developers and organizations to harness the full potential of large language models with ease.
Agentic RAG, Multi-Agent Systems, and Vision Reasoning are three pipelines to find the perfect LLM
Regression testing for AI agents. Snapshot behavior,diff tool calls,catch regressions in CI. Works with LangGraph, CrewAI, OpenAI, Anthropic.
A plugin for AI agent evaluation. Plan evals, generate test cases, interpret results for Copilot Studio agents. Grounded in Microsoft's Eval Scenario Library & Triage Playbook.
Find out where your coding agent starts degrading. Personal context-rot analytics from your own sessions - 100% local, zero-config: uvx contextrot
Omarchy plugin: your wallpaper becomes a live, clickable information desk with every running AI agent (Claude Code, Codex, Grok, Ollama), 7-day heatmap, rate limits, recent prompts, machine stats. Theme-driven.
Mac app and JSON CLI to monitor every Claude Code, Codex, Cursor Agent and Pi session on your machine. See which agent needs permission, which is working, and which is idle. Works with any terminal or IDE. Free, open source (MIT).
Same model, different wrapper: a from-scratch benchmark comparing coding-agent harnesses (codex, pi, opencode, cursor, devin) and open models on correctness, speed, and token cost
CUREBench @ NeurIPS 2025: Benchmarking AI reasoning for therapeutic decision-making at scale
Estimate your GitHub Copilot AI-credit cost in preparation for June 1st.
Observability platform for Digital Employee, providing real-time tracing, session insights, and cost analysis for multi-agent workflows
The Active Reliability Layer for AI Agents. Catch failures, teach fixes, and automate reliability
OpenTelemetry-native SDK for AI agent observability, tracing, evaluation, debugging, governance, and runtime policy enforcement. Framework-agnostic and built for OpenAI Agents, LangGraph, CrewAI, and LLM applications in production.
Benchmark local LLM serving by replaying real agent trajectories
RAGElo is a set of tools that helps you selecting the best RAG-based LLM agents by using an Elo ranker
(THIS IS THE PUBLIC VERSION) Muse-by-Meta usage audit companion designed for those who want to understand Muse’s actions during a multi-step task without pretending to have telemetry they can’t actually access.Check out Muse, your personal AI agent. Code: 5F2V31 https://muse.ai/join
[ICLR 2026] InfoMosaic-Bench: Evaluating Multi-Source Information Seeking in Tool-Augmented Agents
General Purpose Evaluation and Simulation Environment for all your AI related Workflows
[ACL 2026] OPT-BENCH: Evaluating the Iterative Self-Optimization of LLM Agents in Large-Scale Search Spaces
🦞 ClawMark: A Living-World Benchmark for Multi-Day, Multimodal Coworker Agents
10 Claude sessions running. What are they doing? Live dashboard — monitor, cost tracking, search, sub-agent visibility.
gradio region:us
A powerful AI observability framework that provides comprehensive insights into agent interactions across platforms, enabling developers to monitor, analyze, and optimize AI-driven applications with minimal integration effort.
AIRS-Bench: an AI Research Science benchmark for quantifying the end-to-end AI research abilities of LLM agents
AgentCat is an analytics platform for MCP server owners 🐱.
static leaderboard eval:code modality:tools modality:text region:us
OpenTelemetry plugins for every coding agent, in the Agent Plugins format — a marketplace that runs itself: open an issue, the repository validates, reviews, lists and follows your plugin.
Assessing the reliability, accuracy, and safety of large language models, RAG pipelines, and AI agents.
macOS 原生菜单栏 AI 额度监控与本地用量看板 实时追踪 Codex、Claude Code、Antigravity、Cursor 与 Command Code 配额状态,精准分析本地会话 Token 与费用。
Local-first observability your agent reads back. TMA1 records every LLM call, then routes what it sees into the agent's next turn via hooks and MCP.
Token Efficiency For AI Agents
Interactive error analysis skill for AI agents. Studies LLM trace datasets, builds a review UI, monitors annotations, categorizes failure modes, proposes new samples.
A tool for comparing tokenizers
The marketplace for verifiable AI outcomes: state a task, get a fixed price upfront, and Primitive Bench routes across tools to complete it, refunded if the outcome isn't delivered. This repo is the open verification layer.
🤝 The code for "Can Large Language Model Agents Simulate Human Trust Behaviors?"
🏆 The AI coding usage leaderboard — Claude Code, Codex, Gemini CLI & more. Real costs and tokens from ccusage data. Submit with: npx viberank-cli
thinkingbox is a framework for defining tool as MCP servers, running LLM agents against them, and evaluating agent behavior — for offline training-data generation, reinforcement-learning training loops, and model evaluation.
OpenAI Codex CLI session log viewer for JSONL files in ~/.codex/sessions. Browse conversations, tool calls, tokens, collaboration chains, and live sessions on desktop and web.
An open-source profiler for AI agents that analyzes token usage, cost, latency, and optimization opportunities across LLM workflows.
Claude Code Agent Monitoring & Observability on VSCode
AlgoTune is a NeurIPS 2025 benchmark made up of 154 math, physics, and computer science problems. The goal is write code that solves each problem, and is faster than existing implementations.
AgentKernelArena provides an end-to-end siloed-benchmarking environment where different LLM-powered agents—such as Cursor Agent, Claude Code, Codex, SWE-agent, and GEAK—can be evaluated side-by-side on the same GPU kernel tasks, using objective and reproducible metrics.
Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models
Hermes Agent plugin for cron analytics and observability. The dashboard for agentic automations in Hermes.
Anyone can predict the future. Almost nobody keeps the receipt. A hash-chained prediction ledger for LLMs — sealed before the answer exists, graded with proper scoring rules.
Official Code for the paper "Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents "
Code and Data for "MIRAI: Evaluating LLM Agents for Event Forecasting"
Unified realtime agent trace database & search MCP
Best AI API Cost Tracker & Token Usage Monitor for Developers 2026
Token Cost Parity: Multilingual LLM Efficiency Analysis 2026
The online judge for the LLM era. Every case is a real customer delivery: you work it with a coding agent, and your system is judged once it is live.
Initiative to evaluate and rank the most popular LLMs across common task types based on their propensity to hallucinate.
🏰 Real-time dashboard for monitoring and managing Clawdbot AI agents
Official repo of "MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents". It can be used to evaluate a GUI agent with a hierarchical manner across multiple platforms, including Windows, Linux, macOS, iOS, Android and Web.
Litefuse - Agent Observability and Evaluation Platform
gradio leaderboard region:us
Top papers related to LLM-based agent evaluation
Kieker is an observability framework, that consists of an monitoring and tracing agent, and compatible analysis tools
Process monitoring and configuration dashboard for Hermes Agent
Local LLM cost-tracking proxy for OpenAI, Anthropic, Gemini, and pinned OpenRouter calls with token usage, failure, and billing-integrity receipts.
Unified adapter framework for evaluating agent harnesses (claws) on SWE-bench
The KCORES Agent benchmarking project is designed to evaluate the tool-call capabilities of single-modal/multi-modal models.
Benchmark framework that measures cost, quality, and duration of coding agents across any AI Coding Assistant CLI, any model, and any use case — with pluggable verification and real-repo support.
Metrik 可以集中查看本机多个 Agent 的配额余量和 Token 消耗,目前支持 ChatGPT、Claude、GLM、Kimi 等主流 AI 服务。
The NVIDIA NeMo Agent Toolkit UI streamlines interacting with NeMo Agent Toolkit workflows in an easy-to-use web application.
Self-evolving agentic reward framework for image-editing evaluation — 47.4% on EditReward-Bench from only 100 preference demos, no reward-model training. arXiv 2605.08703.
What Effort Level Should I Choose? On the Economics of Reasoning Effort in LLM Agents
Rank LLMs, RAG systems, and prompts using automated head-to-head evaluation