Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 19 of 57
Claude Code session capture and analysis. Trace tool calls, detect backtracks, analyze decisions, track edit chains, and measure plan drift. Local-first, zero-dependency observability.
Live AI judge agent — watches hackathon demos in real-time via Gemini Live API, delivers British-accented commentary via TTS, scores with multi-model ensemble (Gemini + Claude + Groq), detects prompt injection attempts, and runs cross-team deliberation. Built for NEBULA:FOG 2026.
Zero-config visual debugging and auto-evaluation for LLM agents. Local-first tracing with a beautiful dashboard for OpenAI, LangChain, CrewAI, and more.
Comprehensive cost & token usage analyzer for Claude Code with detailed breakdowns by model and token type. Uses ccusage CLI tool and LiteLLM pricing data. The raw token usage data is available in ${HOME}/.claude/projects/<project-name>/<conversation-id>.jsonl files.
Local LLM BenchMarking
Open-source React components for Agentic transparency. Trace every step of AI agent reasoning and make it visible, verifiable, and controllable.
Local-first AI jury skill: 9 seats deliberate, 10 scoring functions audit claims, human final.
The corresponding data and code from our paper "A Benchmark for Deep Information Synthesis" (ICLR 2026). Do not hesitate to open an issue if you run into any trouble!
Chrome DevTools for AI Agents - Real-time debugging, pause, inspect, and step through your AI agent execution. Auto-instruments OpenAI, Anthropic, LangChain.
Mechanistic interpretability workflow platform for activation capture, analysis, interventions, and reports.
Scenario Testing for AI Agents
Four AI engineering skills that gate your agent before it builds. Evaluated with deterministic gates, behavioral probes, and LLM-as-judge rubrics.
BENCHMARKS WITHOUT BORDERS
Lightweight token tracking, cost management, and budget enforcement for LLM API calls
Measure citation and evidence stability in RAG pipelines across runs and configuration changes.
CI-grade evidence and safety tools for AI agents, MCP servers, and upstream contributions
Grafana AI observability plugin for Hermes Agent
Your AI agent skill doctor - 5-dimension scoring + security gate + leaderboard
See what your AI agents do, whether it's worth it, and what it costs - a manager-readable report + local dashboard from Langfuse/LangSmith traces (or a one-line live callback). Open source, runs locally.
Benchmark LLMs on your own codebase. Repo-specific probes, blind rubric-based judging, and correctness-first rankings.
See what your AI coding cost and produced, from the logs already on your disk. Offline, no account.
A framework-neutral taxonomy of agentic-AI artifacts, organized around what matters in production: versioning, inspectability, and governability.
gradio gradio-custom-component custom-component-track adk agent region:us
streamlit region:us
gradio region:us
gradio region:us
git diff for your agent's context window. See exactly what your LLM saw — turn by turn, block by block.
Failure Compiler for AI agents: turn failed outputs into replayable regression cases.
A local Claude Code dashboard for personal analytics: track token usage, API-equivalent cost, sessions, tools, skills, projects, and activity patterns.
Local viewer for VS Code GitHub Copilot Chat sessions. Reads transcript & debug JSONL files locally and shows session metadata, turns, models, tools, agents, token usage and estimated AI costs. No data leaves your machine.
Real-time detection and repair of LLM agent failures — a one-class behavioural monitor at ~200 µs/step, with 2,823 committed traces.
Benchmark harness that compares coding agents — not just models — on real, contamination-resistant tasks: 80 fail-to-pass cases from merged OSS PRs, long-run sequences, SWE-bench-Live, and 25+ domain benchmarks. We benchmark our own agent, octomind, under the same rules.
Deterministic pytest mocks for OpenAI, Anthropic, Gemini, and LangChain—test LLM apps without live API calls.
See what every plugin puts into the model.
Benchmarking search agents for closed-corpus, multi-hop retrieval over visually rich documents.
A local-first multi-agent status island for Windows
DevTools for AI apps. Record, replay, and inspect every prompt, token, tool call, and cost in a visual timeline.
Exact-oracle conformance tests for turn-level credit assignment in agentic RL (GRPO, RLOO, GAE, GiGPO; verl, TRL, OpenRLHF).
看清 AI 编码智能体的每一次可观察行动。 · Trace your agents, down to every tool and MCP.
Measure whether an agent solves the same task the same way twice: path signatures, entropy, divergence, and a minimum-run stability gate.
Trace how a silent tool failure taints a whole agent run: dependency graph, taint paths, and grounded or tainted verdicts.
Agent work that compounds. Turn Claude Code and Codex into repeatable workflows: GitHub-triggered, fully observable, every run scored with evals. Open source and self-hosted.
static trackio trackio-logbook open-experiment icml2026-repro paper-tPcL3fgkjc region:us
Claude & Codex usage tracker for Android: live 5-hour and weekly limits for Claude Pro/Max, Claude Code and ChatGPT Codex, with alerts, a reset planner and home-screen widgets. Free, open source, no servers.
Token usage per skill, subagent and phase for Claude Code: CLI, HTML report, live dashboard and benchmarks.
ThreadBench: Benchmarking Realism in Agentic Simulation of Real-world Discussions (arXiv:2606.14715).
A local-first, provider-agnostic platform to benchmark, score, and diagnose autonomous AI coding agents on real-world polyglot software tasks.
docker region:us
🎛 Memory Intelligence Benchmark (MIB) is an open benchmark for measuring how effectively an intelligent system uses the past to improve future cognition and behavior.
AI Agent Architecture Evaluation Tool — Benchmarked against SOTA Standards// AI Agent 项目架构评价工具 - 对标SOTA 标准
Guidance and xUnit samples for evaluating .NET AI agents with AgentEval — stochastic tests, red-team scanning, trace replay, memory benchmarks, and CI templates
This is the official page of the paper "AssistedDS: Benchmarking How External Domain Knowledge Assists LLMs in Automated Data Science."
Google ADK Evaluation Service(Docker + HTML)
AI agent for log monitoring
Agent cockpit for observing, debugging, and controlling AI agent runs across frameworks, with traces, replay, cost views, and remote-control experiments.
Race a baseline vs a skill or MCP on real tasks. Hard checks show if it got better, faster, or cheaper. Numbers, not vibes.
Token-efficient stochastic testing for AI agents. 5-20x cost reduction. 10 framework adapters. Paper: arXiv:2603.02601
Architectural standards and best practices for building reliable AI Agents and LLM workflows. Defining the framework for AI Reliability Engineering (AIRE).
Head-to-head benchmark comparing the official MCP to the MCP auto-created by Hintas.
Extensible benchmarking suite for evaluating AI coding agents on web search tasks. Compare native search vs MCP servers (You.com, expanding) across multiple agents (Claude Code, Gemini, Droid, Codex, expanding) with automated Docker workflows and statistical analysis.
public ranking system for ai agents
Observantio's Resolver is a ground-up AI engine for modern observability. It correlates metrics, traces, and logs into a unified intelligence layer that detects anomalies, forecasts trends, and automates root cause analysis (RCA).
An SDK for Python AI Agents. Under heavy development.
The open-source testing framework for AI agents. pytest-native, async-first, safety-aware.
Conformance test for stateful AI agents. Measures state correctness over time.
CSR-Bench: Benchmarking LLM Agents in Deployment of Computer Science Research Repositories
LOAB: A benchmark for evaluating LLM agents on end-to-end mortgage lending operations under real regulatory constraints.
⚡ From Zero to Monitoring LLMs in 5 minutes ⚡
Optimizing a Retrieval-Augmented Generation (RAG) system on the CNN/Daily Mail dataset using LangChain, with performance benchmarking and analysis via RAGAS.
The "Storybook" for AI Agents
The open-source run log for MCP agents. Get a shareable report to verify and reproduce what the agent did — plus a protective mode for common dangerous SQL.
The ChatGPT Failures Archive is a repository that collects and documents instances where ChatGPT, a language model developed by OpenAI, does not provide accurate or satisfactory responses. It serves as a valuable resource for researchers, developers, and users interested in understanding the limitations and potential pitfalls of ChatGPT.
Stress/Performance/Speed test utility for ChatGPT or other LLM.
🛟 A fast, colorful TUI to visualize streaming Codex logs in real time
Framework for Post-production Evaluation of LLM based ChatBots
A Gymnasium benchmark suite for evaluating the robustness and multi-task performance of reinforcement learning algorithms in various discrete and continuous environments.
Fully automated LLM evaluator
Zero-config LLM cost & quality monitoring for Node.js - automatically log AI API calls and collect user feedback for analysis using Coolhand.
Explore new metrics and best practices to monitor your LLM systems and ensure safety and quality
Benchmark to evaluate LLM performance in generating ABAP function modules
Benchmark different AI services that can be accessed using Microsoft.Extensions.AI
LLM Benchmarks for everyday business intelligence tasks / company data: Embedding Evaluation. Embedding Classification.
Cost intelligence for Claude Code. Real-time dashboards, pre-flight estimation, budgets, and ghost token auditing.
Demo showing how the Trustworthy Language Model add reliability to LLM outputs and improves RAG, agents, and data enrichment worfklows. can be used to improve fine-tuning of LLMs, accuracy of LLM outputs, and smart routing for RAG and agents.
Benchmark LLMs for your RAG system - Compare Gemini, GPT, Claude & local models (Llama, Mistral, Qwen, Phi) via Ollama with advanced performance metrics.
🤔 wondering if your chunks are good? 🦉 Judie is here to Judge and Evaluate your Chunks! ✨
Terminal-aesthetic OpenClaw Gateway control panel — real-time agent monitoring, billing-aware cost display (subscription/API/local), Subdued Succulent theme, Chart.js telemetry
Diagnose your AI agents in production. Extract policies from prompts, evaluate traces, generate diagnostic reports.
Usage tracking and analytics for MCP servers
The first longitudinal benchmark for AI memory systems. Test LLM agents, RAG pipelines, MCP servers, and memory backends across 1,000 sessions.
Can AI agents do real science? Benchmarking AI agents on KASCADE cosmic ray classification
Comprehensive agent analytics suite for AI agents built with the Google Agent Development Kit (ADK) , LangChain or other popular frameworks, powered by BigQuery Agent Analytics plugin and Grafana
Know what your agents cost. Real-time cost tracking, budget enforcement, and financial observability for AI agents. Zero code changes.
Monitoring and evaluating LLM apps with Langfuse. Presented at PyConZA 2024.
UI interface for Kitaru - durable execution for AI agents, built on ZenML
Eval-first AI agent that triages property maintenance emails. The real work is the eval system around it: trace-driven error analysis, code graders and validated LLM-as-judge (TPR/TNR), component and end-to-end evals, a failure taxonomy, and a CI regression gate. LangGraph, FastAPI, Langfuse.
VS Code extension that shows Claude AI subscription usage (daily & weekly limits) in the status bar. Auto-detects Claude Code credentials. Supports OAuth and enterprise admin keys.
Track your Claude usage and the March 2026 2x promotion — right from your menu bar
Transparent proxy that taps LLM API traffic to extract token usage metrics. ~0.1ms overhead.