Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 2 of 57
Version control for AI agents — track what your agent did, blame any line to a prompt, inspect any step.
🍰 PromptLayer - Maintain a log of your prompts and OpenAI API requests. Track, debug, and replay old completions.
Track AI API quotas across Synthetic, Z.ai, Anthropic (Claude Code), Codex, GitHub Copilot & Antigravity in real time. Lightweight background daemon (<50MB RAM), SQLite storage, Material Design 3 dashboard. Zero telemetry.
Dingo: A Comprehensive AI Data, Model and Application Quality Evaluation Tool
👩⚖️ Agent-as-a-Judge: The Magic for Open-Endedness
The platform for continuously improving AI agents.
Readymade evaluators for agent trajectories
See what your coding agents did and what it cost. Breaks each task down into work steps — tools used, files changed, tests run, time and tokens spent. Local-first dashboard for Claude Code, Codex, OpenCode, and more. No login, no telemetry.
End-to-end Generative Optimization for AI Agents
Real-time observability of claude code sessions & multi-agents.
Visualization and debugging tool for LangChain workflows
Monitor anything in real time
🦞 Official plugin for OpenClaw that exports agent traces to Opik. See and monitor agent behaviour, cost, tokens, errors and more.
🔐 Secure, real-time monitoring dashboard for OpenClaw AI agents. Auth, TOTP MFA, cost tracking, live feed, memory browser and more.
A development-oriented visualization toolkit
Awesome-LLM-Eval: a curated list of tools, datasets/benchmark, demos, leaderboard, papers, docs and models, mainly for Evaluation on LLMs. 一个由工具、基准/数据、演示、排行榜和大模型等组成的精选列表,主要面向基础大模型评测,旨在探求生成式AI的技术边界.
Know how much Claude Code, Codex, Cursor, Copilot and 70+ other AI coding tools you have left — a free, open-source macOS monitor on the edge of your screen.
Trace-native CI/CD for AI agents — production failures become regression tests that block the PR. Auto-detect, cluster, freeze into hermetic cases, replay in CI for $0.
[ICML 2024] TrustLLM: LLM trustworthiness evaluation across truthfulness, safety, fairness, robustness, privacy and ethics. Python/CLI toolkit for local models, compatible APIs and AI agent orchestration.
Measuring how well CLI agents like Claude Code or Codex CLI can post-train base LLMs on a single H100 GPU in 10 hours
MLGym A New Framework and Benchmark for Advancing AI Research Agents
Local analytics dashboard for Claude Code. No cloud, no telemetry.
Comprehensive analytics dashboard for AI coding agents — Cursor, Windsurf, Claude Code, VS Code Copilot, Zed, Antigravity, OpenCode, Command Code
Self-evaluating interview for AI coders
Give AI agents eyes, ears, and verifiable results. Watch Skill turns video, audio and screen activity into searchable, timestamped evidence and proves work with deterministic contracts, not model opinion. DeepWatch is the agent workspace built on DeepSeek Harness. Python + npm, MCP, CLI, REST, Web.
gradio llm perf leaderboard llm performance leaderboard llm performance leaderboard region:us
gradio leaderboard region:us
Chatbot Arena meets multi-modality! Multi-Modality Arena allows you to benchmark vision-language models side-by-side while providing images as inputs. Supports MiniGPT-4, LLaMA-Adapter V2, LLaVA, BLIP-2, and many more!
🌍 AppWorld: A Controllable World of Apps and People for Benchmarking Function Calling and Interactive Coding Agent, ACL'24 Best Resource Paper.
Claude Code & Codex 本地 token 追踪 — 状态栏(Codex 业界首创伪 statusline)、GitHub 风格热力图、多模型成本分析。 | Local token tracker for Claude Code & Codex — status line (industry-first Codex faux statusline), GitHub-style heatmap, multi-model cost analysis
An in-the-wild benchmark for AI agents in the production harness.
Native macOS app to monitor AI usage limits and watch your coding sessions live
Kolo is a text-based Python debugger for AI agents. Capture every executed function call, return value, local variable, HTTP request, and more in greppable trace files.
Data-Driven Evaluation for LLM-Powered Applications
MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers
static region:us
SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
"Unit tests" for your agent skills
Hermy HQ — a self-hostable mission-control dashboard template that pairs with your own local Hermes agent over a Postgres message bus. Send it to your Hermes and it onboards you step by step.
Real-time decoded feed of AI agent actions — monitor multiple Claude Code & Codex sessions, see exactly what each agent is doing, and coordinate swarms efficiently
[ICLR 2026] The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution
VisualWebArena is a benchmark for multimodal agents.
Turn any goal into a short prompt that makes your agent set a real quality bar, run builder and critic pairs, compare blind, and loop until it wins.
MCPMark is a comprehensive, stress-testing MCP benchmark designed to evaluate model and agent capabilities in real-world MCP use.
AutoHarness: Automated Harness Engineering for AI Agents
A lightweight Python SDK and CLI for evaluating recursive self-improvement (RSI) methods across agent benchmarks.
ParseBench - A Document Parsing Benchmark for AI Agents
A beautiful, zero-dependency command center for OpenClaw AI agents
Large language model evaluation and workflow framework from Phase AI.
An Analytical Evaluation Board of Multi-turn LLM Agents [NeurIPS 2024 Oral]
gradio region:us
Fixes prompt cache regression in Claude Code that causes up to 20x cost increase on resumed sessions
The agent responsible for conducting the agent evaluation
Creating realistic RL environments requires collaboration between researchers, engineers, and domain experts across many dimensions: artifacts, environments tools, dynamism of the environment, reproducibility, and more. There is no open source framework for building these environments effectively. Until now.
Measuring and evolving with the frontier of agent work
See your agent think. Zero-config observability & governance for 30 AI agent runtimes: Claude Code, OpenAI Codex, Hermes, OpenClaw & 26 more. Live token costs, sessions, tool calls, crons.
gradio region:us
🦀️ CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents. https://crab.camel-ai.org/
Tool-calling quality benchmark for LLM serving stacks. 80+ deterministic scenarios testing multi-turn orchestration, safety boundaries, and structured output. Supports vLLM, SGLang, and llama.cpp.
Console supporting tracing, prompt engineering and evaluation for Agents developed with Spring AI Alibaba.
RAG evaluation without the need for "golden answers"
A pytest-native safety and security testing framework for agentic AI applications
Server metrics fetching agent, based on SIGAR
pykoi: Active learning in one unified interface
React components for visualizing traces from AI agents
See what every Claude Code and Codex agent on your machine is doing — live, in a little office.
The collaboration layer for teams building AI agents. Test, simulate, and trace your agents, while domain experts annotate behavior and engineers improve the agent from what they find.
A benchmark for evaluating AI agents on realistic business workflows
Open Source LLM toolkit to build trustworthy LLM applications. TigerArmor (AI safety), TigerRAG (embedding, RAG), TigerTune (fine-tuning)
The operational superset of Pi Coding Agent — everything Pi, plus observability, governance, recovery, evaluation and multi-agent orchestration. Pi Coding Agent 的运维超集——完整继承 Pi 的核心能力,并进一步扩展可观测性、治理、故障恢复、评估和多 Agent 编排。
A pixel art virtual office that visualizes your AI agents working in real-time. Claude Code hooks, WebSocket events, and isometric pixel art.
Benchmark for automated failure attributions in agentic systems (🏆 ICML 2025 Spotlight)
Approve and monitor Herdr agents remotely from your phone, a mobile web app for Android/iOS smartphones with push notifications, QR setup, and multi-computer relays over Cloudflare, WebRTC or Tailscale connections.
Token telemetry dashboard for AI autonomous and coding agents — tracks tokens, sessions, tool calls & reasoning across Hermes agent, Claude Code, Antigravity CLI, Codex & more. 100% local.
GoRelic is deprecated in favour of https://github.com/newrelic/go-agent
A curated list of awesome leaderboard-oriented resources for AI domain
Standards for defining and evaluating agent behavior
A generative AI-powered framework for testing virtual agents.
Replay your coding agent sessions as a map and a timeline
AdaRubric: Adaptive Dynamic Rubric Evaluator for Agent Trajectories
Requirement-driven evaluation harness for AI agents and LLM applications. Generate behavior-specific test cases, run them against any target (hosted models, callable wrappers, OTel-traced agents), and inspect local-first artifacts.
AI token usage & quota monitor for the macOS menu bar — native Swift, Liquid Glass, 3D contribution graph. Tracks Claude Code, Codex, Cursor, OpenCode & 25+ agents locally.
Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving
The Golang auto-instrument Agent for Apache SkyWalking, which provides the native tracing/metrics/logging abilities for Golang projects.
[ICML 2025 Oral] Official repo of EmbodiedBench, a comprehensive benchmark designed to evaluate MLLMs as embodied agents.
A comprehensive set of LLM benchmark scores and provider prices. (deprecated, read more in README)
macOS menu bar & Windows tray app pinning Claude Code, Codex, Antigravity & Grok CLI quota, burn rate, and cost to your screen. Local-first, no telemetry. HTML reports, 16 themes.
gradio region:us
🛰 Every AI coding agent on your machine, on one screen — live cost, tokens and tool calls across every provider, and a hold on anything dangerous until you say go. From your desk or your phone.
Zero instrucment LLM and AI agent (e.g. claude code, openclaw, gemini-cli) observability in eBPF
syftr is an agent optimizer that helps you find the best agentic workflows for your budget.
Codes for our paper "ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate"
Sotopia: an Open-ended Social Learning Environment (ICLR 2024 spotlight)
Monocle is a framework for tracing GenAI app code. This repo contains implementation of Monocle for GenAI apps written in Python.
Build, Improve Performance, and Productionize your AI Application
Agent Skills for Langfuse, the open source LLM engineering platform for tracing, prompt management, and evaluation
give agents access to your logs