Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 23 of 57

RAG-Audit-Pipeline★ 4

Diagnose, quantify, and repair the data-quality issues that degrade RAG retrieval accuracy — local-first, no API keys

Elenchus★ 4

Per-claim faithfulness checker for LLM output verifies whether each claim is actually grounded in the source material it was given, with an append-only Verification Log and no default dependency on a real embedding/NLI/LLM API.

AgentSnap★ 4

Deterministic snapshot testing for AI agents

homebrew-agenttap★ 4

Homebrew tap for AgentTap - network-level AI agent observability

quoteBench★ 4

QuoteBench isolates shell quoting and escaping failures in LLM agents across raw, JSON, and wrapped command-transport contracts.

SCOPE★ 4

🔥 Learning When to Trust via Selective Context Preference Optimization

agenterr★ 4

A tiny, dependency-free taxonomy for AI agent tool-call failures: retry, re-prompt, or escalate — with model-ready hints.

openai-agent-sdk-visualizer★ 4

openai agent sdk visualizer

openai-assistants-link★ 4

Link your OpenAI Assistants to a custom store + Evaluate Assistant responses

codex_usage_report★ 4

Go CLI tool to analyze OpenAI Codex session logs and estimate cooldowns

clawtop★ 4

Multi-host TUI dashboard for Anthropic Claude subscription usage. Daemon + viewer split keeps OAuth credentials on your workstation while a tmux dashboard on any host renders the merged view.

ragdrift★ 4

Silent regression detector for RAG pipelines

dsh-lineage★ 4

Content-addressed artifact, fact, action and report lineage for DeepSeek Harness

dsh-capability-receipt★ 4

Content-addressed receipts for skills actually loaded by DeepSeek Harness

hermes-dashboard★ 4

Static system dashboard for a Hermes Agent installation: host status, security, model routing, honest cost, memory, cron economics and a capability map — built by cron from the agent's own telemetry. Stdlib-only Python, EN/RU.

dsh-cot-profile★ 4

Real-time chain-of-thought trajectory profiling for DeepSeek Harness: live wording indicators, profile-family judgment, per-session measurement records

scrim★ 4

The cloth between the tool and the model — tool-output thinning and cost observability for Claude Code and Codex. Fail-open hooks, nothing lost.

control-plane★ 4

Shared HTTP control plane for Chronicle traces and TokenOps spend: one SQLite, one UI. Agents never open the database file.

llm-price-tracker★ 4

A daily-updated reference of LLM model pricing across all major providers (OpenAI, Anthropic, Google, Mistral, and more)

llm_leaderboard_uk★ 4

static region:us

DSH-arena★ 4

Local-first experiment and evaluation workbench plugin for DeepSeek Harness (DSH).

cc-token-meter★ 4

Private, local Claude Code usage intelligence track tokens, estimated cost, cache efficiency, budgets, projects, and live sessions with zero telemetry.

openclaw-multi-agent-test-suite★ 4

OMATS: A reproducible benchmark measuring LLM performance in multi-agent environments. Based on the 5 Stages of Model Capability framework.

dsh-billing-balance★ 4

Community plugins for DeepSeek Harness (DSH) Web GUI: token cost meter, billing balance, update checker — plain JavaScript, no build required.

langshark★ 4

A friendly TUI for inspecting LangChain/LangGraph threads, checkpoints and LLM calls.

llm-cost-profiler★ 4

Track, visualize, and optimize LLM API spending. Monitor OpenAI & Anthropic costs per feature, detect waste, suggest savings. Zero-config Python profiler.

geekclock-claude★ 4

Display Claude.ai usage limits on a $30 desktop WiFi clock (GeekMagic SmallTV ESP32)

dsh-swarm-panel★ 4

Conversation Flow observability and durable multi-agent orchestration for DeepSeek Harness

LTM100★ 4

A multi-user load benchmark for Long-Term Memory (LTM) systems

Human-Intelligence-Assurance-Lab★ 4

Production-grade assurance framework for safe, human-centered AI: evaluation, human calibration, adaptive routing, resilience, and release gates.

picasso★ 4

Local conversation store and read-only dashboard for Microsoft Agent Framework. Two packages, one line each, nothing leaves your machine.

Tether★ 4

Local-first macOS command center for debugging AI agents: capture, visualize, cache, replay, and inspect LLM calls across Codex, Claude Code, OpenAI, and Anthropic.

ccpool★ 4

Know how much of your weekly Claude pool is left, in dollars, and whether you're burning it too fast. A pace-aware budget gauge for Claude Code.

building-the-evaluation-flywheel-pydata-2026★ 4

Companion repo for the PyData Amsterdam 2026 talk Building the evaluation flywheel: how human judgment becomes an evaluator you can run on every change. A data-analysis agent answers 50 Sphere.com business cases, a jury of LLM judges flags where it disagrees, and a human's answers to those boundary questions go into a pass/fail evaluator.

Synathic★ 4

Catch silent failures before your customer does.

environment-harness★ 4

Run agents in persistent shared environments with private observations, checkpoints and auditable evidence.

TokenHound★ 4

Non-intrusive Windows 11 desktop HUD monitoring LLM usage and agent activity across AI coding tools.

ai-halo-valuation-bias★ 4

Context halo in VLM valuation of an identical low-cost necklace across six models (1,164 sessions). Paper: arXiv:2610.34591

DProvenanceKitPython★ 4

Local-first AI reasoning observability for agents: record runs, diff golden baselines, and fail CI on path drift. Complements OpenTelemetry and LangSmith. Python port of DProvenanceKit.

judged-useless-queried-anyway★ 4

Tool-using agents judge a failing source useless, yet keep querying it. Code and data for the paper.

Evals.si★ 4

A highly scalable and pluggable evaluations repository that works with all environments to provide an unified experience for all your evaluation tasks

openevals★ 4

Fast and cheap agent evals. jev as judge.

second-take★ 4

AI reasoning QA + retake note: when an AI answer disappoints, diagnose where its chain-of-thought went wrong and produce a self-contained retake note you paste back, so the original AI answers better. 不满意就诊断那段深度思考哪里错,产出一段可粘回原对话的重拍单(Retake note)。支持文本/图像/视频提示词。Second Take · 诊断与重拍。

ArmBench-LLM★ 4

gradio region:us

agui-inspector★ 4

See the wire of any AG-UI server. Records every request and SSE frame, validates them against the protocol, and drives runs like a client: interrupts, client tools and A2UI surfaces.

JevControl★ 4

Find out whether a smaller model could handle some of your AI agent’s routine choices.

agent-vcr★ 4

Ctrl+Z for AI coding agents: undo anything Claude Code, Codex, Gemini CLI or Cursor did — even rm -rf. Snapshots, bisect, MCP, Agent CI.

pi-usage-live★ 4

Always-on subscription usage card for Pi: see how much of your Claude, ChatGPT Codex, OpenCode Zen, DeepSeek, Kimi, MiniMax, Moonshot, Z.AI and OpenRouter allowance you have left, live in your terminal, without asking.

llm-meter★ 4

Lightweight usage analytics and abuse detection for OpenAI-compatible API gateways

claudecode-switchboard★ 4

Claude Code session dashboard for Mac: live status, token usage, plan limits and messages between sessions. Not affiliated with Anthropic.

evalbot★ 4

EvalBot — local-first chatbot security & quality evaluation (FastAPI + Next.js). Evaluate chatbot answers against your own docs & guidelines with ML/NLP + AI-judge scoring. Apache-2.0.

tokentab★ 4

Local-first LLM cost and quality tracking. Compare cost per successful task without storing prompts.

Spring-AI-Studio★ 4

A full-stack AI benchmarking studio built with Spring Boot 3.4 and React 19. Uses the Spring AI framework to stream and compare multi-LLM outputs (OpenAI, Anthropic, and local Ollama/DeepSeek) with integrated speed benchmarking.

no-hallucination★ 4

Three measured experiments on RAG hallucination: quote-checking, TypeSafe's Jev, and IBM's STAIR. 850+ graded questions, raw responses included.

ai-evo★ 4

Self-hosted Agent quality engineering platform — evaluation as a release gate. 8 eval types with AI judges, datasets, pre-eval scripts, analytics & A/B testing. Gate loop (API keys, CI CLI, webhooks) lands with M1. FastAPI + Vue 3 + Celery.

tiiny-bench★ 4

An independent benchmark for the Tiiny AI Pocket Lab. Measures what the box actually does: prefill scaling, sustained throughput, concurrency and reasoning cost, with NPU budget as the currency. Web app plus a self-contained HTML report. Python stdlib only.

blackgate-ai-game★ 4

AI 长期推理决策 Benchmark|中转站 URL+API Key 全自动跑模型、实时日志、排行榜、赛后复盘与模型画像

model-cost★ 4

Compare LLM API pricing from your terminal. Supports 300+ models across all major providers. https://x.com/saqibameen

ace-monorepo★ 4

Certify AI agents under real chaos — fault injection, LLM trace analysis, statistical hypothesis tests, 12-section PDF report

agent-trajectory★ 4

Turn Claude Code and Codex session logs into an interactive agent trajectory viewer.

arguslm★ 4

The hundred-eyed watcher for your LLM providers. Monitor uptime, TTFT, TPS, and latency across OpenAI, Anthropic, Azure, Bedrock, Ollama, LM Studio, and 100+ providers through a single dashboard. Benchmark, compare, and get alerts — all self-hosted.

vrlbench★ 4

A unified simulator for fair comparison of agent memory systems

xlfp★ 4

Fingerprint LLM API model families and audit provider claims with token-count evidence. CLI with JSON output, request caps, and credential safeguards.

ShortChain★ 4

An observability backend for LLM applications that learns and fit you system

EchoNet★ 4

A benchmark for epistemic arbitration and misinformation resistance in AI agents. Evaluates an agent's ability to distinguish independent evidence from coordinated repetition and manufactured consensus within synthetic social environments.

repro-efficient-inference-for-noisy-llm-as-a-judge-evaluation★ 4

static trackio trackio-logbook open-experiment icml2026-repro paper-tPcL3fgkjc arxiv:2601.05420 region:us

blueclaw★ 3

Observable AI agent runtime with built-in execution tracing. Inspect, replay, diff, and debug agent behavior step by step. Model-agnostic (Claude, Ollama, OpenAI, Gemini). Built on Strands Agents SDK.

ContextSubstrate★ 3

Make AI work reproducible, debuggable, and contestable — like real software.

llmtest-perf★ 3

Production-quality performance validation and regression testing for LLM inference systems

smartness-eval★ 3

🎯 14-Dimension AI Agent Intelligence Assessment | 14维度 AI Agent 智能度自动评估技能 | OpenClaw Skill | Aligned with CLEAR/T-Eval/Anthropic standards

agent-pld-metrics★ 3

PLD: Runtime Phase Model for Stable Multi-Turn LLM Systems

llm-failure-atlas★ 3

A graph-based failure modeling and deterministic detection system for LLM agent runtimes.

PropIA-GenAI-Training★ 3

🏠 GenAI Training Path para QA Engineers — caso de uso PropIA (plataforma inmobiliaria colombiana). Embeddings + ChromaDB, RAG conversacional, agentes ReAct, análisis de contratos con LLMs y evaluación con Precision@K, Recall, MRR y F1. Stack: TypeScript · Anthropic · LangChain.

graphos★ 3

GraphOS is an open-source governance and observability layer designed specifically for LangGraph and the Model Context Protocol (MCP).

Secret_H_Evals★ 3

Multi-agent strategic deception evaluation framework for LLMs using Secret Hitler as a testbed. Analyzes AI reasoning, trust dynamics, and deceptive behavior patterns.

agenttap★ 3

Real-time debugging proxy for Agent2Agent (A2A) multi-agent systems

reagent★ 3

Replay Debugger for AI Agents

quickbench★ 3

🏃 Sovereign Agent Evaluation Framework - Zero cloud dependencies. Local-only, cryptographically signed benchmarks for AI agents. npm run demo = instant evals.

okahu-demo-lmx-travel-agent★ 3

A travel agent python app built with LlamaIndex framework using GPT model from OpenAI to be run locally or on CI/CD with Github Actions

agent-ecosystem-testing★ 3

Empirical evals of agentic docs consumption

agent-observability★ 3

.NET reference implementation for GenAI observability using OpenTelemetry semantic conventions, OTEL Collector, and agent evals

Awesome-AI-Evaluation-Guide★ 3

🤖 Evaluate AI systems effectively with our comprehensive guide to methods, tools, and frameworks for assessing Large Language Models and agents.

exp-agent-performance★ 3

AI agents - performance comparison using major LLMs.

claudedash★ 3

Deterministic, local, passive execution observer for AI agent workflows

moorcheh-benchmarks★ 3

Benchmark tests, experiments and use cases of Moorche

aiopt★ 3

Install-first, local-only LLM cost guardrails: install/doctor/scan → report.md + patches

agent-dash★ 3

Real-time surveillance dashboard for Claude Code Agent Teams — monitor agents, tasks, and messages live

limitai★ 3

CLI tool that monitors LLM rate limit utilization across Claude, Codex, and CLIProxyAPI — see how much of your paid quota you're actually using

rulecatch-sdk★ 3

Official SDK for Rulecatch — AI coding analytics, monitoring, and rule enforcement for Claude Code, Cursor, and AI coding assistants

.github★ 3
laravel-ai-metering★ 3

Production-ready Laravel package for AI usage metering, token tracking, quota management, and Stripe billing integration. Supports OpenAI, Anthropic, and custom providers.

Distill★ 3

See which files burn your LLM tokens — fix them automatically. CI budget gate + MCP server + adapters for Claude, GPT-4o, Gemini, Ollama.

ccmeter★ 3

Local-first spend & cache-efficiency dashboard for Claude Code. Reads ~/.claude/projects, tells you exactly what's costing you. No telemetry, no API key.

promptproof★ 3

Deterministic LLM testing for CI. Record→Replay + policy-as-code to block risky merges (PII, schema drift, regressions, budget creep).

claude-smi★ 3

Terminal UI for monitoring Claude Code token usage and costs

llmtest★ 3

pytest for LLM apps - Test for grounding failures, prompt injection, safety violations, and regressions

brokle★ 3

The AI engineering platform for AI teams. Observability, evaluation, and prompt management for LLMs and AI agents. OpenTelemetry native.

claude-usage-tracker★ 3

🔍 Track and analyze Claude Code usage with rate limit awareness. Parse JSONL logs, calculate costs, and get warnings before hitting rate limits. CLI tool with daily/weekly breakdowns.

pairwiseLLM★ 3

R Package: Pairwise Comparison Tools for LLM-Based Writing Evaluation

concurrent★ 3

The Multi-LLM Benchmarking Tool

More in Agent frameworks & infrastructure