Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 12 of 57
Core engine behind Calibrate, a framework for evaluating AI agents: speech-to-text, text-to-speech, LLM evaluation, end-to-end simulations
Skill RSI is a recursive self-improvement loop for Agent Skills, grounded in evals. Standalone or Codex plugin.
Replay real agent traces through cheaper models to prove which swaps are safe.
[ICML 2026] GUIEvalKit: Open-source Evaluation Toolkit for GUI Agents
流镜 Flowglass — DeepSeek Harness session flowgraph,实时可视化消息、工具组、子代理分支与步骤详情。
A local open-source Frieren desktop companion for monitoring local and remote Pi, Codex, Claude Code and Cursor sessions.
Observability and control plane for AI agents.
[KDD 2026] An automatic, extensible Framework to Evaluate User-Proxy Agents for Human-Likeness. 🌟 Star if you like it!
SWE-Serve: an agentic benchmark of 53 production inference-engineering tasks derived from merged SGLang pull requests, run with Harbor.
Every agent turn becomes a traceable commit carrying the full interaction trace, model, and token cost, plus a live dashboard and a coach that teaches you, from your own sessions, to drive agents and your codebase better.
Agent Reliability Analyzer docs: install, configure and fix agent reliability issues
⚽️🤖 Benchmarking LLMs and deep-research agents on real-world football prediction — from the tactical "who scores in minute 67" to the strategic "who wins the World Cup."
Hermes Agent plugin: transparent per-token cost accounting for local LLM inference rigs
Placebo-controlled benchmark of the most-starred coding agent skills
Animated 3D virtual office for Hermes AgentOS subagents — real-time isometric AI visualization, Three.js.
LoCoBench-Agent: An Interactive Benchmark for LLM Agents in Long-Context Software Engineering
Forensic Observability for AI Agents
The control plane for AI agents. Track, secure, and optimize every agent run.
⚡️ Transform AI/ML operations: Transparency, Control and Cost Optimization. ⚡️
A web-based viewer for Claude Code conversation logs with real-time monitoring and cross-platform support.
docker region:us
Local Windows desktop monitor for your running Claude Code agents - live status, estimated cost, model, host and one-click focus, grouped by project. Read-only and fully offline.
Deterministic, local-only audit reports for Claude Code AI agent sessions. Rust + SQLite. Zero network calls. MCP server for agents.
A self-hosted, mobile-first web GUI for Herdr with browser terminals, workspace and worktree management, file and diff viewers, and AI agent session inspection.
PTCG-Bench: Can LLM Agents Master Pokémon Trading Card Game? (EMNLP 2026 Findings)
htop for local coding agents.
Teach your agent to work with evals: WHEN you actually need an eval or benchmark, HOW to build one that holds up, and how to read what it tells you. Deterministic-first, tool-agnostic.
EC2-Bench (EC²-Bench): benchmark for edge-cloud collaboration in long-horizon LLM agents — task performance, resource usage & privacy exposure. arXiv 2610.17816
Terraform configuration for self-hosting Langfuse on Google Cloud.
OpenTelemetry Benchmark - can AI trace your failed login?
Menu bar app for tracking Claude Code & Codex CLI token usage on macOS — animated, live, SwiftUI
[ICLR 2026] A framework to "create benchmarks" and "evaluate AI co-scientists" in experimental data-driven real-world scientific research.
6 progressive demos to detect, prevent, and self-correct AI agent hallucinations, from Graph-RAG research to production deployment on Amazon Bedrock AgentCore
Invarium is pytest for AI agents. Test behavior, not exact text.
2-in-1 for Claude Code: a live system-tray usage widget (5h/7d plan limits) + local token & cost analytics — per project/model/tool, budgets, forecasts. Unofficial; not affiliated with Anthropic.
The Regression Testing Framework for AI Agents. Replay · Evaluate · Assert · Catch Regressions — in CI. Like Jest for your AI layer.
Portable, hash-verifiable intermediate representation for agent execution trajectories (seals, effects, .tir packages) on top of durable backends like Temporal and DBOS.
A place to evaluate public models
A phone-friendly web interface for Herdr: monitor agents, receive notifications, open real terminals, and send files.
Live token spend dashboard and notifications for Herdr agent panes
Online explainability for LangGraph agents through counterfactual planning and reasoning provenance graphs.
Behavioral regression testing for AI apps and agents: pytest plugin, behavior diff, CI gate, and document extraction metrics
Desktop monitor for coding-agent activity, provider accounts, and infrastructure resources, with a floating companion.
XTrace Desktop is a free Mac app that shows how productive you are with coding agents. It reads the history Claude Code, Codex and Cursor already save on your Mac and turns it into a dashboard: where your hours went, what each session would have cost at API prices and what the outcome was, and how close you are to your usage limits.
Agent decision replay studio - record a decision, rewrite the policy and diff the replay
Help individuals and teams understand their prompting habits and improve how they work with AI coding agents. Get specific insights into prompt clarity, context, rework, outcomes and wasted effort, plus security flags. Bring your own LLM, no SaaS.
Open-source, self-hosted analytics for MCP servers
A recursive self-improvement (RSI) workflow that evolves both target agents (e.g., Pi and DeepSeek Harness) and the RSI system itself.
Fast and intuitive evals on any LLM
Benchmark AI model speeds with TTFT, tokens/sec, and performance metrics .
AI agent evolving strategies through automated self-play overnight. Generic framework with GEPA-inspired feedback loop and Elo tracking.
Windows system tray app for real-time Claude AI usage monitoring
A comprehensive, implementation-focused guide to evaluating Large Language Models, RAG systems, and Agentic AI in production environments.
🔍 一键检测 AI API 服务商是否如实提供所标注的模型 | 支持 OpenAI / Claude / Gemini / DeepSeek 等 9 大模型原厂对比验证
Crash-test insurance claim AI agents before production.
A scope for your Ai coding-agent sessions
docker leaderboard region:us
gradio region:us
gradio region:us
gradio edge llm leaderboard llm edge leaderboard llm edge leaderboard region:us
开源 AI 应用评测平台,支持 RAG、AI Agent、多轮对话、LLM-as-Judge、接口评测、评测报告和人工盲测。Open-source AI evaluation platform for RAG, AI Agents, multi-turn conversations, LLM-as-Judge, endpoint evaluation
A token-spend profiler and cost-regression gate for AI agents.
A local-first desktop dashboard for macOS and Windows that turns Codex CLI and app logs into clear insights on token usage, estimated costs, models, and projects.
Stream Deck plugin showing Claude session/weekly usage
Watch your Claude Code subagents work as a pixel-art office. Rewind any second, see what every agent cost, and turn any session into a timelapse GIF. Read-only, 100% local.
OpenClaw Virtual Office — real-time pixel-art agent office
macOS menu bar app: Claude quota with pace
RL environments with verifiable rewards you can trust. Sandboxed coding and computer-use tasks, verifiers kept apart from the answer, a QA gate against reward hacking, and infra failures kept out of training. Works with verl, TRL, OpenRLHF and Axolotl.
AMBER (sealed-in-amber historical replay evaluation): real-event replay, physical sealing, pre-registered scoring — method spec; cases private, results public
Agent Reliability Analyzer CLI for Windows: find and fix agent reliability issues
Agent Reliability Analyzer CLI for macOS and Linux: find and fix agent reliability issues
Asking for What Was Never Requested: horizontal and vertical proactivity in LLM agents. Need-graph metrics (no LLM judge) and Q&D, which trains a questioner from the consequences of its questions.
Desktop app that shows usage, quotas and cost of every AI coding tool in one panel — Claude Code, Codex, MiniMax, DeepSeek, OpenCode and Kilo — with local session costs, history, forecasts and an optional self-hosted team view.
AgentBehaviorBench (ABB): A benchmark for evaluating AI agents’ ability to test other agents and detect behavioral defects.
Netflix's ontology-driven observability design, rebuilt on 1.9B rows of real operations data and tested claim by claim.
Python SDK for vishwa.ai
Lunary AI Python Client (Analytics, monitoring and evaluations for GenAI apps)
Python decorators to accurately monitor LLM API cost & latency for OpenAI, Anthropic, and other leading AI models
A structured framework for defining, verifying and certifying AI systems.
Token usage tracker for Claude Code and Codex CLI — Rust daemon, 7ms reports, JSONL-parsing
A stop-loss for Claude Code: detects loops, compaction danger, failed tools, and token waste before the next request.
Local dashboard for visualizing usage across seven AI coding agents - sessions, token costs, cache performance, tool calls, and daily breakdowns
Real-time visualization dashboard for Claude Code agents, teams & swarms — isometric pixel world + monitoring dashboard
MindTrial: Evaluate and compare AI language models (LLMs) on text-based tasks with optional file/image attachments and tool use. Supports multiple providers (OpenAI, Google, Anthropic, DeepSeek, Mistral AI, xAI, Alibaba, Moonshot AI, OpenRouter), custom tasks in YAML, and HTML/CSV/JSON reports.
Native macOS menu-bar dashboard for AI subscription quotas, reset forecasts, local token costs, and provider status — Codex, Claude, Gemini, Grok, and more.
LLM / Codex 测速与本机历史分析:TTFT、Token 吞吐、流式曲线,支持 CLI、网页和跨平台桌面端。Benchmark LLM / Codex streaming speed and analyze local Codex history with CLI, web & desktop apps.
gradio region:us
static leaderboard modality:text language:icelandic eval:language region:us
Local AI coding-agent history + API cost analytics (loopback-only).
See where your Claude Code tokens go — cost per call scales 4.3x with context size. Hooks + SQLite + dashboard + MCP server for context window forensics.
Gen AI Evaluation Toolkit on AWS is a flexible, cloud-native accelerator built on AWS serverless architecture that enables comprehensive evaluation of generative AI applications.
Diff your AI agent's behavior between two runs. See exactly which tool calls, args, costs and outputs changed when you swap models or edit prompts.
Caveman prompting, measured. A two-channel evaluation framework for prompt compression and LLM token efficiency: what input vs output compression costs in dollars, accuracy, and whether the reasoning changes. Compress the output not the prompt!! 9 models, 5 benchmarks, 10 languages. EMNLP 2026 Main
Track LLM token usage, enforce rate limit & budget policies, and trigger alerts across OpenAI, Groq, OpenRouter, and AWS Bedrock.
SimuVerity: Benchmarking Agents for Engineering-Grade Simulink Model Generation
Reflect helps engineering teams see whether AI coding agents are actually improving delivery or just burning tokens, failing tools, and creating hidden debt
This repository hosts the instructions and workshop materials for Lab 333 - Evaluate Reasoning Models for Your Generative AI Solutions
Claude Max usage monitor for macOS menu bar
Examples of how to monitor the OpenAI SDK calls using Langfuse