Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 6 of 57
Reproducible voice-AI benchmarking — TTS / STT / S2S latency and accuracy.
AgentVitals Checkup (/checkup) — an AI agent skill that gives your agent a professional health checkup: dual-axis Stability + Welfare scoring, a personality-style title, and a public cross-platform leaderboard. One-line install for Claude Code / OpenClaw / Codex / Coze. Bilingual EN/ZH.
Credential-scoped AI quota, context, and cache in Herdr for Claude, Codex, Grok, Agy, OpenCode, Pi, omp, and Devin.
Benchmark for allocating scarce user attention during long-horizon agent work
A library for benchmarking the Long Term Memory and Continual learning capabilities of LLM based agents. With all the tests and code you need to evaluate your own agents. See more in the blogpost:
A framework for interpreting modern AI systems using Monte Carlo Shapley value estimation. Model-agnostic explainability across language models, vision-language models, video understanding, and autonomous agents.
Evolva 是面向工程场景的本地 Agent 控制台,把模型、工具、记忆、工作流、MCP、Trace、Eval 与 Guardrails 收敛到一个可审计的运行闭环中,让每一次 Agent 执行都可追踪、可验证、可复用、可进化。
Long-horizon arena for end-to-end business agents
Agent Reliability Analyzer: deterministic scanner that finds and fixes AI agent reliability failures
Scope | Agentic Experience evaluation platform (Research preview)
Ship agents you can audit.
Python agent for Opbeat.
Ship agents you can audit.
AgentTrace is a lightweight observability library to trace and evaluate agentic systems.
Local-first Windows tray app for monitoring Claude Code and Codex tokens, costs, sessions, and rate limits
[ICML 26] An evaluation framework assessing long-context retention and long-horizon memory performance for agentic applications (AMA-bench).
Deterministic telemetry & control plane for multi-agent AI systems. Atomic POSIX JSONL event bus, W3C/OpenTelemetry DAG reconstruction, and circular handoff guardrails.
A live fleet console for Claude Code: every session, agent, plan, cost and context window in one browser tab.
We propose a pioneering benchmark to evaluate LLM agents' ability to improve over time in streaming scenarios
Evaluation kit for testing stateful agents
Benchmark self-evolving Agent upon realistic large-scale file workspaces
Cross-platform desktop widget + CLI for real-time Claude Code (and opt-in OpenAI Codex) usage — session/weekly limits, cost, forecasts, burn alerts, 11 themes. One pip install (Linux/macOS/Windows).
AgentBench is an interactive benchmarking platform for tool-using AI agents. Connect your own agent, run practical tasks, and watch every action unfold in real time. With reproducible environments, full execution traces, and deterministic evaluation, AgentBench makes agent performance measurable, explainable, and comparable.
An Evaluation Framework for Self-Evolving-Agent (Loaded Impact on Holdout Final Task)
Visualize your Codex and Claude Code subagents. Advanced traces view, tokens breakdown, and cost usage is also supported.
Open source code of the paper: "OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domain"
Benchmark, evaluate, and optimize skills to ensure reliable performance across all LLMs
Evaluating the Search Agent in a Parallel World
Local-first flight recorder for coding agents : replay every run as a live session map, score the context bill, and write the fix back into AGENTS.md — no API key, one npx command.
gradio leaderboard region:us
gradio leaderboard modality:text eval:generation language:polish judge:auto test:public submission:manual
Testing and observability for multi-agent delegation, MCP tools, permissions, sandboxed actions, prompts, and workflow replay.
Self-hosted dashboard for all your AI plans. Track limits, resets and spend across Claude, Codex, Cursor, Copilot, Grok and more.
Arena: A General Evaluation Platform and Building Toolkit for Single/Multi-Agent Intelligence. AAAI 2020.
Visualize Pydantic AI agent workflows from Logfire traces as an interactive HTML diagram — tools, nested sub-agents, tokens and exact cost.
docker leaderboard region:us
Run-aware token governance for multi-agent systems.
LLM proxy to observe and debug what your AI agents are doing.
[ICML 2026] Evaluating AI Agents on Continuous Software Evolution
Python SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)
Apache SkyWalking Rust Agent
Fast, robust, configurable agent evals
We believe that every SOTA result is only valid on its own dataset. RAGView provides a unified evaluation platform to benchmark different RAG methods on your specific data.
Ready to estimate AI agent work effort using tool-call rounds as the base unit instead of human time anchoring.
📊 Real-time monitoring dashboard for Claude Code agent teams
neatroots: see, check and replay your AI agent's runs on your machine while you build it. Local-first, by Neatlogs.
Fault-injection toolkit for testing the resilience of LLM calls and agentic workflows.
A composable framework for reproducible evaluation of long-running agents.
Record and replay LLM interactions for langchain
Stop reading logs. Start watching them. MermaidTrace is a specialized logging tool that automatically generates Mermaid JS sequence diagrams from your code execution. It's perfect for visualizing complex business logic, microservice interactions, or asynchronous flows.
The GitHub Action for Promptfoo. Test your prompts, agents, and RAGs. AI Red teaming, pentesting, and vulnerability scanning for LLMs. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command line and CI/CD integration.
A coding agent-first CLI for interacting with LangSmith.
Which 5-hour window took you out — and where the money goes. Local profiler for Claude Code, OpenClaw & Hermes Agent: usage windows, burn by source (cron/chats/subagents), the overnight bill, why it burns (loops, retries, idle wake-ups), ready config fixes. Zero deps, nothing leaves your machine.
RL observability for LLM agents: behavior maps, native trainer metrics, rollout investigations, and Python/SQL analysis.
A benchmark for evaluating AI agents on frontier research tasks including system optimization and model development.
Watch what your ai coding agents(claude, codex and opencode) are doing on your terminal and browser.
Codes for paper "SafeAgentBench: A Benchmark for Safe Task Planning of \\ Embodied LLM Agents"
General agent evaluation framework
rook by TestMu AI: Agent Assurance from the terminal. Test how your AI agents actually behave across workflows, tools, and actions, and catch failures and vulnerabilities before they ship.
Self-hosting Langfuse on Amazon ECS with Fargate using CDK Python
Evaluation package that allows benchmarking of agentic AIs from various sources and frameworks by producing statistical results which can be compared across different use cases and datasets.
local-first analytics for AI agent skills
Claude Code Session Dashboard — local observability for ~/.claude sessions
Runs, traces, tool calls, LLM calls, errors, latency, tokens, and eventually detection for agents that appear stuck or caught in loops.
A Benchmark for Evaluating Multi-Hop, Multi-Source Tool-Calling in AI Agents
Secure local trace capture and inspection for Codex and Claude agent sessions.
One-shot LLM eval cases by NAGI STUDIO - same prompt, different agents (model + harness), runnable artifacts side by side.
gradio region:us
ScoutAPM Python Agent. Supports Django, Flask, and many other frameworks.
htop for your AI costs — real-time terminal monitoring of LLM token usage and spending across providers and coding agents
The trust layer for your AI agent on any platform. Trace and evaluate your agent. Full Observability. Run Model portability analysis. Autotune your agent.
🤖 EfficientAI is an open-source chat and voice AI evaluation platform that helps teams test, compare, and ship reliable agents.
【EMNLP 2026 Demo】A debugging framework for agentic AI systems: diagnose failures, attribute root causes, recover with evidence, and validate fixes through reruns.
AI Agent Daily Digest — Collect and analyze your AI coding assistant usage
This project applies the core knowledge from the LLMOps module, including the design and implementation of the API Layer, Inference Layer, Observability Layer, Cache Layer, Guardrails Layer, Routing Layer, and the Data Ingestion Pipeline.
VS Code Extension that shows information about the estimated token usage of GitHub Copilot
Which coding agent wins on real work?
🪢 Auto-generated Java Client for Langfuse API
Agent skills for Arize — datasets, experiments, and traces via the ax CLI
在系統匣同時盯著 Claude、Codex、Antigravity 三家 AI 用量的 Windows 免安裝小工具
Local session-mining loop for staging AI agent improvements and public-safe content ideas
A dishwasher-loading game on a physical simulation of one real machine, built to benchmark language models and playable in a browser. The game inside is called Spotless.
An OpenAI gym environment to evaluate the ability of LLMs (eg. GPT-4, Claude) in long-horizon reasoning and task planning in dynamic multi-agent settings.
Open Source Library for Automated Optimization of AI Agent Workflows
The runtime reliability layer for AI agents.
TraceBrain is an open-source trace management platform for observing, governing, and improving LLM-based agent workflows.
🧪💥 Evaluation framework based on Vitest, the testing framework you familiar with, for agents, models, and more.
LangEvals aggregates various language model evaluators into a single platform, providing a standard interface for a multitude of scores and LLM guardrails, for you to protect and benchmark your LLM models and pipelines.
A web dashboard template for running and monitoring your OpenClaw AI agents. The SHAPE of Max HQ, for anyone building their own mission control.
AxonFlow: Runtime control layer for production AI
LLM evaluation framework for Ruby, powered by RubyLLM. Tribunal provides tools for evaluating and testing LLM outputs, detecting hallucinations, measuring response quality, and ensuring safety. Perfect for RAG systems, chatbots, and any LLM-powered application.
Strands-based agents and harnesses for agentic benchmarks.
skilled: open-source agent skill evaluation CLI and MCP server for Codex and Claude Code. Test skills, detect regressions, and improve with evidence.
RHO: Evolving Agents in the Dark — Retrospective Harness Optimization via Self-Preference. Improving LLM agents from unlabeled past trajectories (arXiv:2606.05922).
Optimize any AI agent’s skills, tools/MCP, and prompts against your own evals.
A privacy-first macOS menu bar app for tracking AI coding quotas, reset times, token usage, and costs across 21+ tools.
Phi-Bench (Φ-Bench): 85 open-source LLM-infrastructure engineering tasks for frontier LLMs & coding agents (KFC/LH/E2E). Self-contained public Dockerfile + offline scoring + reference solution per task. Website: llminfrabench.com
Code for paper AIDABench: AI Data Analytics Benchmark.
XPersona: Evaluating Multilingual Personalized Chatbot