Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 13 of 57

openai-function-tokens★ 20

Predict the exact openai token usage of functions

custom-rag-evals★ 20

Applying domain specific evaluations to RAG chunking and embedding functions

agentbreak★ 20

infrastructure chaos to test the resilience of ai agents

agentrial★ 20

Statistical evaluation framework for AI agents

shekel★ 20

LLM budget control and cost governance for AI agents. Python library for token budgets, usage limits and guardrails for OpenAI, Anthropic, LangChain, LangGraph and agentic systems.

riskkernel★ 20

Deterministic cost / loop / time budgets · full observability · crash-resumable runs · human-approval gates · a memory you own. Self-hosted. Your keys. No telemetry. Point it at your existing agents - one env var.

pulsedeck★ 20

PulseDeck — the reporting inbox for your AI agents.

precisionMemBench★ 20

Precision-aware retrieval benchmark for LLM memory systems.

compare-llms★ 20

gradio region:us

subquadratic-llm-leaderboard★ 20

gradio region:us

leaderboard★ 20

gradio region:us

AgentsScope★ 20

Lightweight observability for multi-agent AI pipelines. Self-hosted, no cloud dependency — add two lines of Python and see real-time agent events, DAG graphs, token usage, and session history on a local dashboard.

output-drift-financial-llms★ 20

Reproducible benchmarks for LLM output drift and agent replay in financial operations: DFAH-Bench, the installable dfah package, and a hands-on workshop.

coding-agent-leaderboard★ 20

gradio leaderboard region:us

agentagon★ 20

Evidence-backed audits, evaluations, and measured improvements for any AI agent.

CorruptChain★ 20

Trace how a silent tool failure taints a whole agent run: dependency graph, taint paths, and grounded or tainted verdicts.

jev-rag-benchmark★ 20

Reproducible benchmark for measuring Jev reranking quality, latency, and cost in RAG

whileai-sdk★ 20

Scientific RL and SFT post-training for AI agents: build evals that can fail, simulate situations, judge checked against people, pass@1 with a 95% interval, train with GRPO/SFT/DPO, prove every gain on a held-out set. uv add whileai

tessary★ 20

Open-source agent reliability platform for discovering, investigating, and fixing production agent failures

attest★ 20

Evidence-grounded evaluation for AI agents — verifies each claim against the agent's real tool outputs (constrained, evidence-grounded model judgment, not holistic LLM-judge guesswork), with confidence intervals.

Claude-Detection★ 20

检测 Claude API 中转 / 代理是否掺水(模型替换、偷换、降智、注入、截断)的 Claude Code Skill 与参考实现

RAG-Performance★ 19

Measuring RAG solutions throughput and latency

liquefy-openclaw-integration★ 19

Liquefy is a local-first OpenClaw vault system for packing, verifying, searching, and auditing agent runs with domain-aware compression, policy/redaction, and proof artifacts.

llm-observability★ 19

技術書典#17 - 『俺たちと探究するLLM Observabilityアプリケーションのオブザーバビリティ』で使用するサンプルアプリケーション

ChatGPT-Token-Usage-Pre-Calculator★ 19

Perfect for anyone who needs to quickly calculate the token amount of ChatGPT in prompts for their project.

claude-o-meter★ 19

CLI tool to extract Claude AI usage metrics as JSON for status bars like HyprPanel. Includes Nix flake & Home Manager module.

realign★ 19

Realign is a testing and simulation framework for AI applications.

nuclia-eval★ 19

Library for evaluating RAG using Nuclia's models

evret★ 19

Evals framework for Information Retrieval Systems

HumanStudy-Bench★ 19

HumanStudy-Bench: Towards AI Agent Design for Participant Simulation

agent-belt★ 19

Reproducible evaluation for AI coding agents. Multi-turn scenarios against Claude Code, Codex, Copilot, Cursor, Gemini CLI, Goose, OpenCode, or any custom agent you plug in; verify behavior with rule checks, workspace diffs, multi-judge LLM consensus; pin reliability with pass^k variance across trials. Git worktrees, optional Docker sandbox.

Multi-Agent-Benchmark-Tool★ 19

Interested in running a conspiracy of agents (technical term) on your Local AI Infra? Who isn't!

claude-code-metrics-prometheus★ 19

Grafana dashboard for monitoring Claude Code CLI usage on Prometheus-compatible backends (Prometheus, VictoriaMetrics, Mimir, Thanos). Consumes OpenTelemetry metrics emitted via OTLP.

Agent-Behavior-Lab★ 19

A controlled-experiment lab for studying how tool-using LLM agents behave — vary tools, personas, and conversation histories, run repeated trials across models, and measure the effect on safety and tool-call behavior.

aura★ 19

Agent Usage Reporter & Analyzer

replayd★ 19

Turn failed AI agent runs into replayable regression tests. Catch regressions before you ship.

Chatgpt-token-counter★ 19

A Chrome extension that tracks your daily ChatGPT token usage with a live progress bar, model detection, and plan-aware limits.

llm-usage-tracker★ 19

macOS desktop + always-on Docker app that monitors Claude and OpenAI/Codex subscription quotas and tracks every Claude Code & Codex agent session in real time — tool calls, touched files, token usage, and analytics. Local-first: SQLite, SSE streaming, encrypted credentials, no telemetry.

ai-cost-cutter-skills★ 19

10 skills that cut your AI bill with proof instead of vibes. MIT, works in Claude Code, Codex, Cursor. Every skill's check is executed by CI.

evalcore★ 19

Snapshot testing for LLM apps and agents, built to run locally and block regressions in CI.

hermes-jekyl-hyde★ 19

Adversarial LLM Reversal for hermes-agent

obslab-llm-observability★ 19

Search for a holiday and get destination advice from an LLM. Observability by Dynatrace.

rag-observatory★ 19

Research prototype for trace-based observability and failure analysis in retrieval-augmented generation.

typed_evals★ 19

Fast, typed Python framework for Jev-as-a-Judge/Decision-model evaluation for LLMs, RAG, and AI agents along with human-label calibration.

BAGEN★ 19

BAGEN (Budget-Aware Agent): benchmark and training code for budget-aware LLM agents that estimate token, time, and money cost mid-task.

loongsuite-js-plugins★ 18

OpenTelemetry instrumentation plugins for JavaScript-based AI coding agents. Collect traces, tool calls, and LLM metrics from Claude Code and OpenClaw — zero code changes required.

spent★ 18

Claude Code session cost tracker. Efficiency score, productive vs wasted breakdown, live terminal dashboard.

logfire-assistant★ 18

AI-powered tool that helps you debug, analyze, and understand your applications using Pydantic Logfire.

llm-registry★ 18

A centralized registry for discovering and managing LLM model capabilities. Track model features, costs, and limitations across providers like OpenAI, Anthropic and other open source models. Supports both verified model definitions and user-managed entries with local storage.

trustgate★ 18

Black-box AI reliability certification via self-consistency sampling and conformal calibration

ai-validator★ 18

ai-validator is a powerful library that helps to extract and validate structured data from the output text of language models.

openclaw-langfuse★ 18

OpenClaw plugin for Langfuse LLM observability — traces every agent turn with sessions, token usage, latency, and cost tracking. Zero dependencies, drop-in install.

llmstat★ 18

CLI-first, local LLM cost tracker for indie devs — drop it in front of any OpenAI-compatible API and watch token usage + spend in real time. No cloud, no signup.

state-harness★ 18

Runtime safety net for LLM agents. Detects token spirals, kills doomed tasks early, tells you exactly why. Rust core, Python SDK. pip install state-harness

mastra-vs-langchain★ 18

The same AI agent pipeline built in Mastra and LangChain. Runs in parallel, measures everything.

llm-leaderboard★ 18

streamlit region:us

Agents_Perspective★ 18

gradio region:us

turnstile★ 18

Session-layer circuit breaker for LLM & agent traffic — a transparent streaming proxy that meters every session and kills the one that goes rogue, without taking down your app.

minesweeper★ 18

A Minesweeper benchmark for LLM agents. One identical board, up to nine models in parallel, one clock, one tool layer.

freeinference_agentic_trace★ 18

Agent serving trace from https://freeinference.org

tokentop★ 18

Terminal dashboard for monitoring Claude Code, Codex, OpenRouter, and VelociRouter API usage in real time

speculum★ 18

monitor for a local LLM runtime (llama-swap, NInfer, Strata, more) -- metrics ported to berth.

clawforge★ 17

One dashboard for all your AI assistants

ocel-generator★ 17

Generate realistic multi-agent workflow traces with LLM-enriched content, semantic validation, and PM4Py compatibility. pip install open-agent-traces

OpenTrace★ 17

A local reverse proxy that records every LLM request/response to SQLite. No cloud, no data leaving your machine.

LLM-Hallucination-Detection-Script★ 17

A comprehensive toolkit for detecting potential hallucinations in LLM responses. Compatible with any LLM API (OpenAI, Anthropic, local models, etc.)

multiagent-visibility-tool★ 17

DevTools for multi-agent AI — trace, visualize and debug agent workflows in real time

api-model-spy★ 17

🔍 Detect API model dilution and fingerprint which LLM is actually running behind any OpenAI-compatible or Anthropic endpoint

attest★ 17

Evidence-grounded evaluation for AI agents — verifies each claim against the agent's real tool outputs (constrained, evidence-grounded model judgment, not holistic LLM-judge guesswork), with confidence intervals.

smartcomment★ 17

A lightweight Python tracing toolkit for recording execution graphs from existing systems.

auto-bench-audit★ 17

Automated auditing pipeline for LLM and agent benchmarks — surfaces task ambiguity, environment conflicts, and evaluation bugs.

claumon★ 17

Claude Code dashboard: see where your limits are headed before you hit them. Calibrated usage forecasts, live rate-limit gauges, session costs, memory browser. Single binary, zero config, plus a VS Code extension.

token-meter★ 17

Get insights into token usage for Claude code and Codex sessions with a dashboard and menu bar widget for mac

open-agent-leaderboard★ 17

gradio region:us

LLM-Inference-Benchmark★ 17

gradio region:us

open-japanese-llm-leaderboard-v2★ 17

gradio 日本語 Japanese leaderboard language:日本語 language:Japanese region:us

empirical_agent★ 17

An end-to-end framework for running, sandboxing, and scoring agentic LLMs on complex data-science and econometric replication tasks.

rag-ci★ 17

Regression testing and configuration sweeps for RAG pipelines, with the statistics to know whether a change actually helped.

vfs-bench★ 17

The Virtual Filesystem Interface for AI Agents

codex-usage-badge★ 17

A lightweight Windows badge for Codex usage limits, reset times, and reset credits. Tauri 2 / Rust. 余量

openclaw-deck★ 16

Self-hosted dashboard for OpenClaw AI agents. Cost tracking, budget enforcement, session replay, config safety.

rasa-model-report★ 16

Simple open-source Rasa command-line add-on that generates training model health reports for your projects. 📈🔍🧾🤖🧠

llm-fighter★ 16

Evaluate LLM agentic capabilities through combat games

goose★ 16

Goose is a Python library for comprehensive, repeatable testing of LLM agents and their tools through conversational scenarios, with a built-in UI for visualization.

claudetrack★ 16

Real-time token tracking, cost forecasting, and rate limit prediction for the Anthropic Claude API. Self-hosted, open source, free forever.

impermanent★ 16

Impermanent: a live benchmark for forecasting that measures temporal generalization under real world data drift. The benchmark includes foundation models (from AWS, Google, NXAI, Salesforce), classical statistical methods, and simple baselines. Metrics such as MASE and scaled CRPS, along with championship style points, are published on the site.

NiceEval★ 16

build eval for your agent in 10 mins

OR-Space★ 16

OR-Space: a full-lifecycle workspace benchmark for industrial optimization agents

eval-harness★ 16

Typed evaluation for OpenCode skills and AI agents: capability, regression and product checks; provider-neutral evidence grading, provenance, and CI reports.

causal-agent-replay★ 16

Counterfactual replay and causal attribution over LLM-agent trajectories: find which step actually caused a failure by intervening on it.

llm-rank-themselves★ 16

gradio region:us

lmsys-chatbot-arena-leaderboard★ 16

gradio leaderboard region:us

AgentXRay★ 16

Web dashboard for viewing AI agent session logs — supports OpenClaw, Codex, and Claude Code | 多平台 AI Agent 日志查看面板

openagent-eval★ 16

Local-first evaluation framework for RAG systems and AI Agents. 18+ metrics, CLI + SDK, framework-agnostic. The pytest of AI evaluation.

claude-pulse-android★ 16

Android home screen widget to track your Claude AI usage. Sideloadable APK.

Cribble★ 16

The world leaderboard ranking for AI usages.

BLINDSPOT★ 16

Live Benchmark for long Horizon tool using Agentic Systems

epochflow★ 16

可观测、可评测、可部署的多 Agent 工程平台:结构化路由、Skills、工具/RAG、50 模型实验中心与 LLM-as-Judge。FastAPI + Vue。

FrontierVideo★ 16

FrontierVideo: Benchmarking Coding Agents for Professional Video Production

RAG-BenchLab★ 16

Build RAG evaluation testsets from documents and benchmark RAG systems.

More in Agent frameworks & infrastructure