Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 11 of 57

openclaw-tool-call-viewer★ 29

🦞 Web UI for browsing OpenClaw session tool call history. Zero dependencies, just Node.js.

agentprobe★ 29

Tool-agnostic TypeScript library for observing coding agent sessions. Built-in Cursor provider, extensible to Claude Code, Codex, and OpenCode.

third-eye★ 29

See where your AI coding money goes. Self-hosted dashboard for Claude Code, Cowork, and Codex CLI — all data stays on your disk.

RagQALeaderboard★ 29

RAG-QA Leaderboard

AdamBench★ 29

Benchmark of local LLM models for agentic-coding daily tasks.

BFCL-CN★ 29

Berkeley Function Calling Leaderboard (BFCL) with Chinese-Language Evaluation

llmops_langchain★ 29

LLMOps framework using LangChain, Pinecone and Kubeflow

context-scope★ 29

A local-first macOS debugger for LLM context windows, token usage, tool calls, and agent execution.

ai-agent-benchmark★ 29

AI coding agents comparison - 80+ agents, SWE-Bench leaderboard, pricing. Devin, Cursor, Claude Code, Copilot, and more. December 2025.

AEGIS★ 29

AEGIS: Automated Error Generation and Attribution for Multi-Agent Systems

persian_llm_leaderboard★ 29

gradio region:us

rl-dashboard★ 29

docker agent-collab region:us

leaderboard★ 29

gradio region:us

hermes-skilleval★ 29

Verification-gated skill routing and self-improvement harness for Hermes-style agent skills

burnly★ 29

Track your AI coding-agent token burn across Claude Code, Codex, OpenCode, Antigravity, and more.

ml-mmtoolsandbox★ 29

MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents

anthropometer★ 29

The agentic debugger for Claude Code, Codex CLI and Gemini CLI: a token usage, cost and context-window tracker that shows what the tokens ARE — live memory map, per-file tokens, cache economics, compaction autopsies, PDF reports.

rungraph★ 29

One command turns every agent session already on your disk into a graph: subagents, tool calls, retries, the moments you said no. Ask your agent what happened and the graph lights up the nodes it means. npx rungraph: no hooks, no setup, all local. Works with Claude Code, Codex, Hermes, opencode and Cursor.

boundary-bench★ 29

BoundaryBench: benchmark coding-agent harnesses (Claude Code, Codex, Terminus 2, Grok) on Terminal-Bench tasks under hardened NIST-derived sandbox policies. Inspect AI + Harbor compatible.

overseer-dashboard★ 29

Overseer — pixel-accurate agent-operations dashboard (React, TypeScript, Vite)

Awesome-AI-Evaluations-Tools★ 29

Collection of frameworks and tools for AI evalations, including tool-use, agentic AI, MCP, and multimodal

AgentProbe★ 28

Drop-in pytest plugin for regression-testing AI agents — snapshot baselines, semantic comparison, mock LLMs

MATEval★ 28

MATEval is the first multi-agent framework simulating human collaborative discussion for open-ended text evaluation.

leaf-playground★ 28

A framework to build scenario simulation projects where human and LLM based agents can participant in, with a user-friendly web UI to visualize simulation, support automatically evaluation on agent action level.

clancy★ 28

HTTP/HTTPS proxy with real-time monitoring for Agent traffic. Designed for debugging and monitoring Claude Code, Anthropic API calls, and other AI agent communications.

langsmith-fetch-skill★ 28

🔍 AI observability skill for Claude Code. Debug LangChain/LangGraph agents by fetching execution traces from LangSmith Studio directly in your terminal.

self-care★ 28

Agent trace analysis and context remediation plugin for Claude Code. Detects quality issues in your AI agent traces — goal drift, hallucinations, missed actions, and more.

meta-harness-on-islo★ 28

Meta-harness optimization loop wired onto Islo sandboxes. POC: 0/5→5/5 in four proposer steps. Built on islo.dev.

llm.log★ 28

Know what you spend, see what you send. Lightweight local proxy that logs every LLM call - costs, tokens, full prompts and responses.

llm-colosseum★ 28

gradio arena region:us

wikiracing-llms★ 28

docker region:us

FlowLens-AgentOps★ 28

Observable diagnostics, failure attribution, metrics, and no-key replay for multi-agent runtimes.

usageBar★ 28

macOS 菜单栏工具,聚合显示多个 AI 编程/Agent工具(Claude Code / Codex / Cursor / Qoder / 悟空/ Workbuddy 等)的 token 用量,全本地直读零配置

observra★ 28

Observra is an open-source observability and telemetry framework designed for AI and agentic systems, providing deep visibility into agent behavior, execution flows, runtime events, and inter-agent communication.

check-usage★ 28

Native macOS widget for AI usage limits — Claude, Cursor, Codex, Copilot, Gemini, Grok. No extra login. No telemetry.

Realm★ 28

See your AI agents work. Real-time 3D visualization of Claude Code activity, multi-agent orchestration, and REST API integration for external systems.

General-AgentBench★ 28

Benchmark Test-Time Scaling of General LLM Agents

bpf-benchmark★ 28

AI Agent eBPF optimization benchmark and framework

llm-evaluation-system★ 28

Agentic AI-guided evaluation system for comparing LLMs with multi-judge jury scoring

HarnessFix★ 28

This is the official repository for the paper From Failed Trajectories to Reliable LLM Agents: Diagnosing and Repairing Harness Flaws

agent-eval★ 28

CI for LLM agents — change the prompt, swap the model, edit a tool, refactor the loop: replay the eval suite and let statistics decide if anything got worse. OTel tracing, calibrated LLM-as-judge,adversarial simulated users, and a GitHub Actions gate that blocks significant regressions.

decrypto★ 28

Implementation of the Decrypto benchmark for multi-agent reasoning and theory of mind.

collaborative-intelligence★ 28

A benchmark for AI agents as teammates. Paper: https://arxiv.org/abs/2611.02633

repoagentbench★ 27

SWE-bench for your codebase — mine your merged PRs into local, contamination-free coding-agent benchmarks. Adapters: claude-code, aider (Opus 4.7 / GPT-5.5 / Sonnet 4.6 / Gemini 3.1 Pro).

newrelic-nestjs-integration★ 27

Sample dockerized NestJs framework service integration with New Relic Node.js Agent showing how to instrument the app in order to use all the Agent’s features (like Service Maps or Distributed Tracing)

agiflow-sdks★ 27

LLM QA, Observability, Evaluation and User Feedback

agentest★ 27

Vitest for AI agents. Scenario-based testing with simulated users, tool-call mocks, and LLM-as-judge evaluation. Lives in your project like Playwright.

Toybox★ 27

The Machine Learning Toybox for testing the behavior of autonomous agents.

Polaris★ 27

Data collection and evaluation framework for on-device agents

claw-agent-dashboard★ 27

A powerful web-based dashboard for managing OpenClaw AI agents. Monitor system health, browse workspaces, manage sessions, edit blueprints, and interact with agents in real time.

harness-investigations★ 27

A repository for collecting LLM agent challenges, hypotheses / methods, and experimental results. For science!

openai-api-mock★ 27

A Node.js module for mocking OpenAI API responses in a development environment

RAG-evaluation-harnesses★ 27

An evaluation suite for Retrieval-Augmented Generation (RAG).

llm-usage-dashboard★ 27

一次看到 Codex (OpenAI) 與 Claude Code (Anthropic) 的訂閱用量、額度與花費的本機桌面儀表板。安裝: npx llm-usage-dashboard

llm_contamination_detector★ 27

gradio region:us

llm-auto-eval★ 27

gradio region:us

DeepTraceHub★ 27

RedSearcher's framework for deep search agent trajectory synthesis, QA filtering, and model evaluation, supporting ReACT and DeepSeek-style agent loops.

TRACE-Bench★ 27

TRACE Bench: Task-driven Roleplay Agentic Checklist Evaluation

evalhub★ 27

An LLM Benchmark evaluation platform for automated model testing and analysis. 面向大语言模型的 Benchmark 自动化评测平台,支持模型测试与结果分析

hermes-best-models★ 27

LLM evals for Hermes agents. Benchmark to find the best models for Hermes

agentic-llm-token-and-performance-tests★ 27

The same HTTP API in 10 languages at 3 sizes, 30 implementations. Token cost for LLM agents, plus cold start, latency, throughput, CPU and memory. All measured, all verified over live HTTP.

self-evolve★ 27

Methodology and deterministic evaluation tools for iterative improvement of skills, repositories and agent workflows, with independent review and regression checks.

trustabl-vscode★ 27

Agent Reliability Analyzer extension for VS Code and Cursor: find agent reliability issues

tokenmeter★ 27

Claude Code token usage and cost dashboard. Cost per prompt, cache misses and usage limits for Claude Code, Codex and GitHub Copilot CLI, read from local logs. Zero dependencies, nothing leaves your machine.

chronicle★ 27

Record-and-replay for agent decision graphs: reproduce a prod agent failure as a committed regression test, and re-run your fix without live LLM calls.

when-agents-rule★ 27

A long-horizon RTS benchmark: LLM agents govern rival civilizations and pick their road to victory - raise a Wonder in peace or raze every rival. Browser-only, zero build step.

hermes-bench★ 27

Local benchmarking UI for LLMs and AI agents

epi-recorder★ 26

EPI (Evidence Packaged Infrastructure) packages AI execution as evidence.

entire-dashboard★ 26

A web-based analytics and visualization dashboard for data generated by entireio/cli

ClaudeGauge★ 26

Real-time Claude API usage monitor on ESP32-S3 with an LCARS Star Trek display — tracks costs, tokens, rate limits, and Claude.ai subscription status

budi★ 26

Local-first cost analytics for AI coding agents. See where your tokens and money go across Claude Code, Cursor, and more.

loongsuite-js★ 26

OpenTelemetry instrumentation plugins for JavaScript-based AI coding agents. Collect traces, tool calls, and LLM metrics from Claude Code and OpenClaw — zero code changes required.

indic_llm_leaderboard★ 26

streamlit region:us

de-arena★ 26

gradio region:us

FlowBench★ 26

[EMNLP 2024] FlowBench: Revisiting and Benchmarking Workflow-Guided Planning for LLM-based Agents

evaluatorq★ 26

Evaluation module for the orq platfrom in python, including Red Teaming and Agent Simulation

awesome-eo-agent-benchmarks★ 26

A curated list of benchmarks for evaluating Earth-Observation (EO) AI agents

Realm★ 26

See your AI agents work. Real-time 3D visualization of Claude Code activity, multi-agent orchestration, and REST API integration for external systems.

2Wild-Coding-Agent-Latency-Monitor★ 26

Live dashboard: fire N parallel streaming coding-agent runs at any OpenAI-compatible endpoint — per-run TTFT/tok-s/E2E, real kill switch, per-GPU metrics, shareable summary. Zero deps.

frieren-agent-monitor-pet★ 26

Animated macOS desktop pet and AI coding agent monitor for Claude Code, Codex, Cursor, and Pi, with local and remote SSH session tracking.

basically-agent-evals★ 26

Your Evals Are Lying To You: a hands-on TypeScript curriculum for LLM evals, from a five-line assertion to a live diagnostic console. By Tejas Kumar.

tokentray★ 26

Claude Code, Codex, Gemini (Antigravity), Cursor, Grok & OpenCode usage limits live in the Windows taskbar — session & weekly limits, reset times, pace warnings.

ai-usage★ 26

Native macOS menu bar and Windows 11 system-tray apps for tracking remaining Claude, Codex, and GitHub Copilot usage.

Tokens-4-Breakfast★ 26

Track AI usage, spend, subscriptions and rate limits across 11 providers (Claude Code, Claude, Codex, Cursor, Copilot, OpenAI, Anthropic API, Grok, DeepSeek, Mistral, OpenRouter) in your Mac menu bar. Local-first, no telemetry.

tavily-web-eval-generator★ 25

A Tavily-based agent generating evaluation datasets for web search RAG systems.

clawatch★ 25

Open source observability for OpenClaw agents. Real-time monitoring, cost tracking & management, and smart alerts. Free forever.

pathlight★ 25

Visual debugging, execution traces, and observability for AI agents.

AzureOpenAILogProbs★ 25

Examples of how-to use Azure OpenAI Log Probabilities (LogProbs) feature to enhance Generative AI - Q&A grounding.

vibe_cockpit★ 25

Real-time monitoring dashboard for AI coding agent fleets: session health, output streaming, and observability across Claude, Codex, and Gemini

AIBattery★ 25

A battery style usage meter for Claude Code and OpenAI Codex. Rate limits, context health, and token usage in your macOS menu bar.

errex★ 25

Lightweight, self-hosted Sentry alternative. Rust + SQLite, ~10 MB RAM. Sentry-SDK compatible ingest, fast SvelteKit dashboard, MCP-ready for AI agents.

PixelClaw-Office★ 25

Pixel-art OpenClaw agent office dashboard with demo mode and configurable sprites. A fun dashboard for multi-agent health monitoring.

VS-Bench★ 25

VS-Bench: Evaluating VLMs for Strategic Reasoning and Decision-Making in Multi-Agent Environments

WorldMemArena★ 25

Official codebase for the paper "WorldMemArena: Evaluating Multimodal Agent Memory Through Action–World Interaction"

ARGUS★ 25

Catch silent failures in AI agents before your users do

agentlens★ 25

Open-source observability and audit trail platform for AI agents. MCP-native, tamper-evident event logging, real-time dashboard.

ccgauge★ 25

Claude Code & Codex Dashboard — local web UI for Cluade & Codex token usage and cost. npx ccgauge

crm_llm_leaderboard★ 25

gradio region:us

babysitter-observer-dashboard★ 25

Real-time observability dashboard for AI agent runs: live task progress, journal events, and orchestration state. Published on npm.

opik-claude-code-plugin★ 25

Log Claude Code sessions to Opik, the open-source LLM observability and evaluation platform, built by Comet. Tracing, evaluation, and skills for observable AI applications.

More in Agent frameworks & infrastructure