Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 9 of 57
Open-source self-hosted web tool for evaluating Agent Skills with rubric scores, Deep Review, and improvement suggestions.
Learn How To Observe, Manage, and Scale, Agentic AI Apps Using Azure AI Foundry - with this hands-on workshop
Evaluation for AI apps and agent
llmBench is a high-depth benchmarking tool designed to measure the raw performance of local LLM runtimes (Ollama, llama.cpp) while providing deep hardware intelligence.
A library for evaluating Retrieval-Augmented Generation (RAG) systems (The traditional ways).
[EMNLP 2025] Dataset and Code of "PersonaGym: Evaluating Persona Agents and LLMs"
Unified workbench for building and debugging AI Agents
PinchBench is a benchmarking system for evaluating LLM models as OpenClaw coding agents. Made with 🦀 by the humans at https://kilo.ai
Lightweight cost observability for every API call in your stack. Track LLM tokens, database ops, and external services from your terminal.
Agent observability and guardrails for OpenClaw — risk scoring, audit trails, dashboard.
LLM spend tracking and budgets for Rails — by user, feature, or any tag, in your own database, no proxy.
Local-first CLI for benchmarking LLMs on real hardware — quality, speed, reliability, and a real multi-turn agent loop.
Runtime provenance for AI and scientific workflows—capture, enrich, and query workflow data via observability adapters and code annotation across edge, cloud, and HPC.
CoDA-Bench is a benchmark for code agents on data-intensive tasks. 🎈代码智能体能搞定数据密集型任务吗?
Benchmark and demo harness for measuring LLM agent tool-context budget tradeoffs.
Local dashboard for Claude Code, Codex, OpenCode, OpenClaw and Qoder session history, token usage, cost tracking, tool calls and subagent replay.
htop for AI Agents — monitor token usage, costs & tool calls across Claude Code and Codex in real time
Taste-Bench: measuring the long-horizon judgment of LLM agents at real decision forks
A web dashboard for monitoring and orchestrating AI coding agents across your devices.
Ruby gem for calling observability APIs
Multiprocess agent plugin for Fluentd
Decision orchestration and reconciliation for AI changes.
Agent-based OpenTracing instrumentation in Java
ML-Dev-Bench is a benchmark for evaluating AI agents against various ML development tasks.
An n8n community node that brings Langfuse observability to your OpenAI chat workflows.
Code for "APTBench: Benchmarking Agentic Potential of Base LLMs During Pre-Training"
Your agents do the work — you keep the knowledge. A local-first dashboard (+ desktop pet) that records and replays your Claude Code, Codex CLI and Cursor sessions: timeline, AI session summaries, analytics — all on your machine.
Adversarial, deterministic benchmarking for AI agent skills — measures whether a skill makes the agent measurably better. Includes biomedical eval cases.
This is an AI agent to compare the baselines of different type of AI chips. It simplifies the work of AI product managers.
Trace Compare & Live Maze for DeepSeek Harness: visualize agent exploration (main path, detours, backtracks) from session logs or live sessions
Create and improve AI agents that get better over time with automatic optimization and continuous learning
General Agent Benchmark for OpenClaw, made by Qwen Team, Alibaba Group.
Elastic APM Android Agent
Diagnose the performance of your RAG🩺
Public Goods Game (PGG) Benchmark: Contribute & Punish is a multi-agent benchmark that tests cooperative and self-interested strategies among Large Language Models (LLMs) in a resource-sharing economic scenario. Our experiment extends the classic PGG with a punishment phase, allowing players to penalize free-riders or retaliate against others.
Bring Galileo powered eval workflows into Claude Code and Codex.
Native Swift AI token usage monitor for the macOS menu bar — track Claude Code, Codex, Cursor, Copilot and other AI coding agent usage with a cute 2D/3D contribution graph.
Multi-Agent LLM Evaluation Docs: https://maseval.readthedocs.io/
Learning to Evaluate Before Improving: Automatic Rubric Induction for Automatic Research Agents
Native macOS menu bar app that tracks local AI coding agent token usage like daily steps.
Visualize AI context & traces.
Rubric-driven evaluation for AI agents. Executable criteria, code and model judges, and inspectable trace reports.
Drop-in LLM cost tracker — change one import line, see where your money goes. Supports OpenAI, OpenRouter, Azure, Ollama.
Visualize your ChatGPT usage with interactive charts and insights.
AI agent conversation history browser
Data and materials to reproduce Bloomberg's investigation into racial and gender bias in OpenAI's GPT
AI lifecycle platform for classical ML and agentic systems. Versioned, encrypted registries for data, models, experiments, prompts, agents, MCPs, and skills. Pre/deploy/post-deployment patterns. Real-time monitoring and evaluation via Scouter. Rust core, governed by design
An agentic system for autonomously generating explainable and reproducible time-series anomaly detection rules using LLMs.
Official Repository for our paper: PlanBench-XL: Evaluating Long-Horizon Planning of LLM Tool-Use Agents in Large-Scale Tool Ecosystems
Peek into your agents. Real-time visualization for multi-agent AI systems.
AI-driven toolkit that automates evaluation processes for AI agents
gradio region:us
ARIES is an open-source experimentation framework for agent serving systems research.
Windows port of TokenBar — a tray monitor for AI coding-agent token usage, cost, and subscription quota. WinUI 3 shell on the same Rust parsing core.
Track your Claude API usage from your MacBook's notch
Workflow, Pipeline & AI Agent Observability Platform. Metrics, logs and traces for all your automations.
Mission control for your AI agents 🎮
Benchmark for evaluating LLM agents in bioinformatics
Lightweight eval framework for MCP servers, built on mcp-agent
Live Claude, Codex, Antigravity, DeepSeek and opencode-go usage limits in your Mac's notch.
static region:us
Benchmark and demo harness for measuring LLM agent tool-context budget tradeoffs.
[NeurIPS 2026] CoffeeBench: Benchmarking Long-Horizon LLM Agents in Heterogeneous Multi-Agent Economies
OpenTelemetry integration for AI Agents Cursor / Copilot / Gemini / Claude / Codex that records agent activity and exports detailed telemetry with OTLP protocol
Compact AI agent runtime control plane with governed tools, HITL approval, resumable execution, trace-driven evaluation, and SWE-bench-shaped evidence.
Trace OpenAI Codex CLI coding sessions, agent turns, tool calls, token usage, and subagents with Langfuse for comprehensive observability.
Lightweight, process-scoped energy and carbon instrumentation for Python applications, AI models, and LLM pipelines.
Comprehensive evaluation of 15 AI coding agents (Cursor, Copilot, Claude, Replit, v0, Warp, etc.) with implementations, screenshots, and professional scoring. Published on Turing Post.
A framework for evaluating function calls made by LLMs
Tune your AI Agent to best meet its KPI with a cyclic process of analyze, improve and simulate
LangChain, Llama2-Chat, and zero- and few-shot prompting are used to generate synthetic datasets for IR and RAG system evaluation
Redefine agentic evaluation in enterprise AI. By simulating realistic scenarios, these evals enables rigorous, multi-dimensional assessment of LLM-powered productivity agents.
Observable AI cognition in Kotlin Multiplatform. Every agent decision emits a structured event.
CLI for agent builders to benchmark & chaos test your AI Agents. Text, Voice, Code supported.
Social leaderboard for LLM token usage — sync your agent usage with the CLI, climb the leaderboard
Test harness for voice agents. Import from Retell, VAPI, Bland, LiveKit. Run autonomous simulations. Evaluate with LLM judges.
Open-source test harness for AI agents. Stress-test production agents with adversarial multi-turn scenarios in CI
Desktop cockpit for DeepSeek Harness (dsh): token usage & cost tracking, budget alerts, runtime auto-update with rollback, Quick Ask hotkey, scheduled tasks, session search. Win+macOS. DeepSeek Harness 桌面驾驶舱:成本/用量监控 · 自动更新 · 定时任务
Mission Control dashboard for monitoring and managing AI agent fleets. Track costs, performance, context health, security posture, and market intelligence.
Open-source runtime attribution and context observability for data access by AI agents and applications
A Rust TUI and Desktop app for exploring AI coding tool token and cost usage
gradio leaderboard region:us
Behavioral testing for AI agents.
Dokimi (Greek δοκιμή, "test, proof") measures whether a skill, MCP doc source, agent profile or repository guidance actually helps a coding agent: paired on/off arm comparisons, novel-vs-known content analysis, and repository scans with graded agent runs.
Personal usage analytics for Pi coding agent. Temporal-aware insights, context-aware suggestions, model efficiency analysis.
C^3-Bench: The Things Real Disturbing LLM based Agent in Multi-Tasking
A comprehensive framework for benchmarking single and multi-agent systems across a wide range of tasks—evaluating performance, accuracy, and efficiency with built-in visualization and tool integration.
OpenClaw 贾维斯看板是一个面向 OpenClaw 多 Agent 系统的实时作战指挥中心。 它提供系统健康、Agent 运行态、任务管理与多模型用量看板(MiniMax / OpenAI / Gemini / GLM)的统一视图,支持本地 API、实时刷新与可视化配置,帮助你在一页内快速完成监控、诊断和操作。
LLM-Blender: Ensembling framework that maximizes LLM performance via pairwise ranking. Employs PairRanker to rank candidates and GenFuser to merge outputs, generating superior responses by combining the diverse strengths of multiple open-source models.
Know how much Claude Code your friends are burning through. Create a group, share a code, compare usage.
Benchmark Everything Everywhere All at Once, a fully autonomous agentic system for benchmark construction and customization.
PAST-Bench: Benchmarking the Foundations of Recursive Self-Improvement in Personal Agents
Overnight experiment: can an AI agent make money, and can you trust what it reports? The agent produces claims; a separate verifier it cannot invoke produces the facts.
Benchmark coding agents and agent harnesses on 226 deterministic file-editing tasks with byte-exact verification and a public leaderboard.
A framework to capture, analyse and export agentic sessions
PTXBench: an open-source environment to evaluate GPU kernel languages for LLM agents
An alerting layer for agentic frameworks
Detect hallucinations in LLM responses. Verify every claim against source documents using hybrid STS + NLI. Works with LangChain, LlamaIndex, or any RAG pipeline. pip install longtracer
Hermes Agent-focused architecture and runtime health checks
Standalone observability dashboard and developer playground for the Laravel AI SDK. Zero build steps. Telescope for your AI agents.