Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 7 of 57
Self-hosted browser tool for counting and inspecting text tokens, with a Chinese UI adapted from the OpenAI Tokenizer.
CATArena is an engineering-level tournament evaluation platform for Large Language Model-driven code agents (LLM-driven code agents), based on an iterative competitive peer learning framework.
Terminal neurovisualizer for Hermes Agent. 100+ animated themes that react to what your agent is doing. Watch your AI think.
Pixel-art virtual office for AI agent teams
A desktop app for head-to-head coding agent battles, with single-agent mode and free inference. Pick the winning agent and continue from its work.
Self-hosted observability for AI coding agents. Clone. Configure. See.
Watchtower — monitor, inspect, and debug all API traffic between AI coding agents (Claude Code, Codex CLI) and their APIs, with a real-time web dashboard.
AgentX python SDK. Build multi-agent AI workforce. Run evaluation. Trace your agent. Full Observability.
Open source tooling and scoring framework to measure how well services work for AI agents.
TypeScript SDK for MCP server observability, built on OpenTelemetry. Gain insight into agent usage patterns, contextualize tool calls, and analyze server performance across platforms. Integrate with any OpenTelemetry ingest service including the Shinzo platform.
AI gateway and observability server written in Rust. Designed to help optimize multi-agent workflows.
Proxy Claude Code Requests through LiteLLM to Arize
Dashboard for monitoring and managing Hermes agent sessions, tool usage, and activity in real-time
Optimize LLM agent harnesses through tunable Formulas to dynamically enhance the agent, and improve those Formulas with optimizers based on collected agent rollout trajectories.
An unofficial local visualizer for Jev (TypeSafe): a live view of every call your code makes. Not affiliated with TypeSafe AI.
Deterministic agent reliability detection rules for the Trustabl Analyzer
A deterministic telemetry framework for evaluating oversight-sensitive behavior in LLM agents.
Live view of your AI agents at work.
🖥️ macOS-native dashboard for OpenClaw AI agents — monitor, control, and optimize your agent from a sleek web interface
DataSciBench: An LLM Agent Benchmark for Data Science (Findings of ACL 2026)
Context meters, quota tracking, and provider rate limits for agents running in Herdr.
Local Codex quota, Credits, and token analytics inside ChatGPT.
Herdr World — visualize agent work in Office, Tree, Graph, and Spaces; control Herdr agents across local and SSH hosts.
Observability for AI coding assistants — cost tracking, efficiency scoring, and anti-pattern detection for Claude Code, Cursor, Copilot, and more. Local-first; New Relic optional.
Thoughtbox is an intention ledger for agents. Evaluate AI's decisions against its decision-making.
End-to-end Langfuse workshop using a TypeScript Agent to teach the AI engineering loop: tracing, prompt management, monitoring, datasets, experiments, and evaluation.
Benchmark for comparing agent harnesses on everyday online tasks — fixes the base model, varies the harness. Sister project of ClawBench, same scoring pipeline.
TUI monitoring dashboard for Hermes AI agent
Open-source Environment toolkit of claw-like agents, support task/harness generation and evaluation
Real-time cost observability for Model Context Protocol (MCP) tool calls. Wraps any MCP server, attributes spend per tool/project/customer. Free tier 25K calls/mo. EU-hosted
Prompting tests for evaluating LLMs, coding agents, and agentic models
docker region:us
🐼 Steer your LLMs and agents at the token level: A web tool for token visualization & control, model inspection, installation-free harness, and LLM & agent trajectory annotation.
GenoTEX: An expert-curated benchmark for evaluating LLM agents on real-world gene expression analysis tasks. (MLCB 2025 Oral)
A toolbox for benchmarking Multimodal LLM Agents trustworthiness across truthfulness, controllability, safety and privacy dimensions through 34 interactive tasks
LLM and agent evaluation for Java & Kotlin. Runs in JUnit and CI. Spring AI, LangChain4j, Koog, Embabel, and any LLM client.
An open-source Python SDK for analyzing, evaluating, and curating agent traces stored in BigQuery. Built on top of the BigQuery Agent Analytics, it provides a consumption-layer toolkit for agent observability, analysis, evaluation, and advanced capabilities like context graph at scale.
DairyBench: A Long-Horizon Multi-Agent Benchmark for Business Decision-Making
Reference architecture, contracts, skills, and fixtures for safe evaluator-driven self-improving ecosystems.
Agent reliability rulebook: the threat model and rationale behind each Agent Reliability Analyzer rule
Can LLMs beat classical HPO? A benchmark comparing classical, LLM-based, and hybrid methods on Karpathy's autoresearch.
Personal usage analytics for Pi coding agent. Temporal-aware insights, context-aware suggestions, model efficiency analysis.
Benchmark suite for evaluating agent and LLM memory systems
Is your AI coding spend delivering? Offline-first analytics for Claude Code, Codex CLI, Gemini CLI, GitHub Copilot CLI and Cline — $/100 AI lines per project, every verdict carrying its own confidence, a self-contained dashboard, exec plugins in any language, and a self-hosted team server. No telemetry; prompts are never read.
Evaluate Multimodal LLMs as Embodied Agents
Datadog for AI Agents.
A holistic framework for advancing LLMs as data science agents
Terrarium: Multi-turn data engine for evaluating and optimizing LLM agents in living environments.
Agent-as-a-Judge grading framework for evaluating AI outputs/deliverables
Track what every AI coding agent costs and see the limit coming. Per-session, per-model cost for Claude Code, Codex, Cursor and 37 more tools from your provider's own token counts, plus limit gauges, a dangerous-command guard and dashboard terminals. Free, Apache-2.0, runs on your machine.
Agent dashboard runs reliability scans, skills, and network access probes.
🔍 AI-powered diagnosis for Scikit-learn models: Detect overfitting, data leakage, class imbalance & more with LLM-generated insights
Benchmarking the gap between AI agent hype and architecture. Three agent archetypes, 73-point performance spread, stress testing, network resilience, and ensemble coordination analysis with statistical validation.
A/B test whether pre-gathered context helps coding agents (Claude, Codex, Cursor, Grok) complete tasks faster, cheaper, and better
Comprehensive AI Model Evaluation Framework with advanced techniques including Temperature-Controlled Verdict Aggregation via Generalized Power Mean. Support for multiple LLM providers and 15+ evaluation metrics for RAG systems and AI agents.
Cubicle: a live pixel-art office for your AI agents (Paperclip, Claude Code, Codex, Gemini CLI). See who works, who needs you, who failed; KPI board, Telegram alerts, HD themes. Zero dependencies, read-only.
Multi-Agentic System Observability SDK
An Extended Benchmarking of Multi-Agent Reinforcement Learning Algorithms in Complex Fully Cooperative Tasks
A comprehensive benchmarking framework for evaluating and optimizing CPU-centric agentic AI systems across multiple workloads, reproducing results from the research paper: "A CPU-Centric Perspective on Agentic AI"
This repository is the official implementation of ZSC-Eval: An Evaluation Toolkit and Benchmark for Multi-agent Zero-shot Coordination. Pre-trained Agent Zoo: https://huggingface.co/Leoxxxxh/ZSC-Eval-policy_pool.
Открытый бенчмарк LLM: какая нейросеть лучше пишет код 1С:Предприятие (BSL). Объективная оценка LLM по методике SMOP с реальным исполнением в 1С — Claude, GPT, Gemini, DeepSeek, YandexGPT, GigaChat.
gradio leaderboard region:us
OpenAI Codex Desktop in-app usage monitor for Windows — tracks subscription quota/reset time, token usage, API accounts and API keys via local CDP injection. No app.asar patching.
Enterprise Grade, Voice AI simulation SDK for testing your AI Agents
MemEye: A Visual-Centric Evaluation Framework for Multimodal Agent Memory
Agent 始终无法命中缓存巨浪费Token? 来看 PromptXRay! PromptXRay 是一个代理工具,只需要接入到AI API 调用中间, 就可以分析 OpenAI 兼容API的 Prompt 缓存行为
Runic macOS Menu Bar App for AI Usage Cost and Token Tracking
Comprehensive AI Model Evaluation Framework with advanced techniques including Temperature-Controlled Verdict Aggregation via Generalized Power Mean. Support for multiple LLM providers and 15+ evaluation metrics for RAG systems and AI agents.
CLI to monitor your program's OpenAI API token usage.
The Python library with command line tools to interact with Dynabench(https://dynabench.org/), such as uploading models.
Automated LLM Coding Tournaments. There can be only one (winning code solution from the competing AIs)
A native macOS menu bar app that shows your Claude.ai usage at a glance — colour-coded icon, live counter, reset timer. No API key needed.
gradio region:us
Can LLM/MARL agents coordinate in long-horizon, open-ended tasks? NeurIPS 2026 (Evaluations and Datasets).
Agent Skills Evaluation Framework
Minimize AI and Agent failures into minimal reproducible traces.
Open-source observability for your LLM application.
Official code for TimeSeriesGym: A Scalable Benchmark for (Time Series) Machine Learning Engineering Agents
🤖 A curated list of resources for testing AI agents - frameworks, methodologies, benchmarks, tools, and best practices for ensuring reliable, safe, and effective autonomous AI systems
Iterative agent harness improvement: run a coding agent on a hard task, generate the reusable tooling it was missing, qualify it, and replay fresh sessions with it activated. Works with Codex and Claude Code.
Agent token and task monitoring, thousands of pet avatars, cross-platform support, and compatibility with popular AI agents. 一眼查看 Token 与任务状态,海量 Petdex 形象可选,支持 Windows/macOS/Linux、ESP/Wio Terminal,兼容 Cursor、Codex、Claude、Windsurf 等主流 Agent。
Grade every agent, scraper and model in a claim's chain — provenance, trust scoring and audit evidence for LLM pipelines
Orbit: A framework for multi-agent safety and security evaluations
Measure and safely cut the token cost of LLM tool and MCP schemas, with equivalence checking and a CI gate.
A toolkit for developing AI agents, including agent-debugger: Terminal debugger for LangGraph & LangChain agents. Debug LLM agents with state inspection, tool calls, semantic breakpoints, and Python program stepping in one Textual UI.
Implementation of 12 AI agents evaluation techniques
Real-time .NET proxy and dashboard for inspecting AI coding agent API calls (currently supports Claude Code)
MiroEval: A benchmark and evaluation framework for deep research agents — 100 tasks (70 text, 30 multimodal) assessed across synthesis quality, factuality, and research process. 13 systems evaluated.
AgentCat is an analytics platform for MCP server owners 🐱.
A local resource sentinel for the multi-agent era — monitors RAM, processes & disk used by AI agent CLIs (Claude Code, Codex, MCP servers), flags leaks/zombies/runaway caches, and proposes AI-driven cleanup you approve.
static leaderboard llm downloads open-models korea analytics region:us
We track and analyze the activity and performance of autonomous code agents in the wild
A lightweight, general-purpose harness for tool-using LLM agents, fair benchmark evaluation, harness baselines, and personal assistant workflows.
gradio region:us
Source code for our paper: "Put Your Money Where Your Mouth Is: Evaluating Strategic Planning and Execution of LLM Agents in an Auction Arena"
Claude, ChatGPT, Cursor & Copilot usage limits in your macOS menu bar: reset countdowns, pace warnings, zero setup
Open-source AI model evaluation and benchmarking framework for LLMs (OpenAI, Ollama, Claude, Gemini)
See your LLM generation speed (tokens/second) after every agent turn.
Registry-as-ground-truth verification & autonomous research framework for agent-driven development.