Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 23 of 57
Diagnose, quantify, and repair the data-quality issues that degrade RAG retrieval accuracy — local-first, no API keys
Per-claim faithfulness checker for LLM output verifies whether each claim is actually grounded in the source material it was given, with an append-only Verification Log and no default dependency on a real embedding/NLI/LLM API.
Deterministic snapshot testing for AI agents
Homebrew tap for AgentTap - network-level AI agent observability
QuoteBench isolates shell quoting and escaping failures in LLM agents across raw, JSON, and wrapped command-transport contracts.
🔥 Learning When to Trust via Selective Context Preference Optimization
A tiny, dependency-free taxonomy for AI agent tool-call failures: retry, re-prompt, or escalate — with model-ready hints.
openai agent sdk visualizer
Link your OpenAI Assistants to a custom store + Evaluate Assistant responses
Go CLI tool to analyze OpenAI Codex session logs and estimate cooldowns
Multi-host TUI dashboard for Anthropic Claude subscription usage. Daemon + viewer split keeps OAuth credentials on your workstation while a tmux dashboard on any host renders the merged view.
Silent regression detector for RAG pipelines
Content-addressed artifact, fact, action and report lineage for DeepSeek Harness
Content-addressed receipts for skills actually loaded by DeepSeek Harness
Static system dashboard for a Hermes Agent installation: host status, security, model routing, honest cost, memory, cron economics and a capability map — built by cron from the agent's own telemetry. Stdlib-only Python, EN/RU.
Real-time chain-of-thought trajectory profiling for DeepSeek Harness: live wording indicators, profile-family judgment, per-session measurement records
The cloth between the tool and the model — tool-output thinning and cost observability for Claude Code and Codex. Fail-open hooks, nothing lost.
Shared HTTP control plane for Chronicle traces and TokenOps spend: one SQLite, one UI. Agents never open the database file.
A daily-updated reference of LLM model pricing across all major providers (OpenAI, Anthropic, Google, Mistral, and more)
static region:us
Local-first experiment and evaluation workbench plugin for DeepSeek Harness (DSH).
Private, local Claude Code usage intelligence track tokens, estimated cost, cache efficiency, budgets, projects, and live sessions with zero telemetry.
OMATS: A reproducible benchmark measuring LLM performance in multi-agent environments. Based on the 5 Stages of Model Capability framework.
Community plugins for DeepSeek Harness (DSH) Web GUI: token cost meter, billing balance, update checker — plain JavaScript, no build required.
A friendly TUI for inspecting LangChain/LangGraph threads, checkpoints and LLM calls.
Track, visualize, and optimize LLM API spending. Monitor OpenAI & Anthropic costs per feature, detect waste, suggest savings. Zero-config Python profiler.
Display Claude.ai usage limits on a $30 desktop WiFi clock (GeekMagic SmallTV ESP32)
Conversation Flow observability and durable multi-agent orchestration for DeepSeek Harness
A multi-user load benchmark for Long-Term Memory (LTM) systems
Production-grade assurance framework for safe, human-centered AI: evaluation, human calibration, adaptive routing, resilience, and release gates.
Local conversation store and read-only dashboard for Microsoft Agent Framework. Two packages, one line each, nothing leaves your machine.
Local-first macOS command center for debugging AI agents: capture, visualize, cache, replay, and inspect LLM calls across Codex, Claude Code, OpenAI, and Anthropic.
Know how much of your weekly Claude pool is left, in dollars, and whether you're burning it too fast. A pace-aware budget gauge for Claude Code.
Companion repo for the PyData Amsterdam 2026 talk Building the evaluation flywheel: how human judgment becomes an evaluator you can run on every change. A data-analysis agent answers 50 Sphere.com business cases, a jury of LLM judges flags where it disagrees, and a human's answers to those boundary questions go into a pass/fail evaluator.
Catch silent failures before your customer does.
Run agents in persistent shared environments with private observations, checkpoints and auditable evidence.
Non-intrusive Windows 11 desktop HUD monitoring LLM usage and agent activity across AI coding tools.
Context halo in VLM valuation of an identical low-cost necklace across six models (1,164 sessions). Paper: arXiv:2610.34591
Local-first AI reasoning observability for agents: record runs, diff golden baselines, and fail CI on path drift. Complements OpenTelemetry and LangSmith. Python port of DProvenanceKit.
Tool-using agents judge a failing source useless, yet keep querying it. Code and data for the paper.
A highly scalable and pluggable evaluations repository that works with all environments to provide an unified experience for all your evaluation tasks
Fast and cheap agent evals. jev as judge.
AI reasoning QA + retake note: when an AI answer disappoints, diagnose where its chain-of-thought went wrong and produce a self-contained retake note you paste back, so the original AI answers better. 不满意就诊断那段深度思考哪里错,产出一段可粘回原对话的重拍单(Retake note)。支持文本/图像/视频提示词。Second Take · 诊断与重拍。
gradio region:us
See the wire of any AG-UI server. Records every request and SSE frame, validates them against the protocol, and drives runs like a client: interrupts, client tools and A2UI surfaces.
Find out whether a smaller model could handle some of your AI agent’s routine choices.
Ctrl+Z for AI coding agents: undo anything Claude Code, Codex, Gemini CLI or Cursor did — even rm -rf. Snapshots, bisect, MCP, Agent CI.
Always-on subscription usage card for Pi: see how much of your Claude, ChatGPT Codex, OpenCode Zen, DeepSeek, Kimi, MiniMax, Moonshot, Z.AI and OpenRouter allowance you have left, live in your terminal, without asking.
Lightweight usage analytics and abuse detection for OpenAI-compatible API gateways
Claude Code session dashboard for Mac: live status, token usage, plan limits and messages between sessions. Not affiliated with Anthropic.
EvalBot — local-first chatbot security & quality evaluation (FastAPI + Next.js). Evaluate chatbot answers against your own docs & guidelines with ML/NLP + AI-judge scoring. Apache-2.0.
Local-first LLM cost and quality tracking. Compare cost per successful task without storing prompts.
A full-stack AI benchmarking studio built with Spring Boot 3.4 and React 19. Uses the Spring AI framework to stream and compare multi-LLM outputs (OpenAI, Anthropic, and local Ollama/DeepSeek) with integrated speed benchmarking.
Three measured experiments on RAG hallucination: quote-checking, TypeSafe's Jev, and IBM's STAIR. 850+ graded questions, raw responses included.
Self-hosted Agent quality engineering platform — evaluation as a release gate. 8 eval types with AI judges, datasets, pre-eval scripts, analytics & A/B testing. Gate loop (API keys, CI CLI, webhooks) lands with M1. FastAPI + Vue 3 + Celery.
An independent benchmark for the Tiiny AI Pocket Lab. Measures what the box actually does: prefill scaling, sustained throughput, concurrency and reasoning cost, with NPU budget as the currency. Web app plus a self-contained HTML report. Python stdlib only.
AI 长期推理决策 Benchmark|中转站 URL+API Key 全自动跑模型、实时日志、排行榜、赛后复盘与模型画像
Compare LLM API pricing from your terminal. Supports 300+ models across all major providers. https://x.com/saqibameen
Certify AI agents under real chaos — fault injection, LLM trace analysis, statistical hypothesis tests, 12-section PDF report
Turn Claude Code and Codex session logs into an interactive agent trajectory viewer.
The hundred-eyed watcher for your LLM providers. Monitor uptime, TTFT, TPS, and latency across OpenAI, Anthropic, Azure, Bedrock, Ollama, LM Studio, and 100+ providers through a single dashboard. Benchmark, compare, and get alerts — all self-hosted.
A unified simulator for fair comparison of agent memory systems
Fingerprint LLM API model families and audit provider claims with token-count evidence. CLI with JSON output, request caps, and credential safeguards.
An observability backend for LLM applications that learns and fit you system
A benchmark for epistemic arbitration and misinformation resistance in AI agents. Evaluates an agent's ability to distinguish independent evidence from coordinated repetition and manufactured consensus within synthetic social environments.
static trackio trackio-logbook open-experiment icml2026-repro paper-tPcL3fgkjc arxiv:2601.05420 region:us
Observable AI agent runtime with built-in execution tracing. Inspect, replay, diff, and debug agent behavior step by step. Model-agnostic (Claude, Ollama, OpenAI, Gemini). Built on Strands Agents SDK.
Make AI work reproducible, debuggable, and contestable — like real software.
Production-quality performance validation and regression testing for LLM inference systems
🎯 14-Dimension AI Agent Intelligence Assessment | 14维度 AI Agent 智能度自动评估技能 | OpenClaw Skill | Aligned with CLEAR/T-Eval/Anthropic standards
PLD: Runtime Phase Model for Stable Multi-Turn LLM Systems
A graph-based failure modeling and deterministic detection system for LLM agent runtimes.
🏠 GenAI Training Path para QA Engineers — caso de uso PropIA (plataforma inmobiliaria colombiana). Embeddings + ChromaDB, RAG conversacional, agentes ReAct, análisis de contratos con LLMs y evaluación con Precision@K, Recall, MRR y F1. Stack: TypeScript · Anthropic · LangChain.
GraphOS is an open-source governance and observability layer designed specifically for LangGraph and the Model Context Protocol (MCP).
Multi-agent strategic deception evaluation framework for LLMs using Secret Hitler as a testbed. Analyzes AI reasoning, trust dynamics, and deceptive behavior patterns.
Real-time debugging proxy for Agent2Agent (A2A) multi-agent systems
Replay Debugger for AI Agents
🏃 Sovereign Agent Evaluation Framework - Zero cloud dependencies. Local-only, cryptographically signed benchmarks for AI agents. npm run demo = instant evals.
A travel agent python app built with LlamaIndex framework using GPT model from OpenAI to be run locally or on CI/CD with Github Actions
Empirical evals of agentic docs consumption
.NET reference implementation for GenAI observability using OpenTelemetry semantic conventions, OTEL Collector, and agent evals
🤖 Evaluate AI systems effectively with our comprehensive guide to methods, tools, and frameworks for assessing Large Language Models and agents.
AI agents - performance comparison using major LLMs.
Deterministic, local, passive execution observer for AI agent workflows
Benchmark tests, experiments and use cases of Moorche
Install-first, local-only LLM cost guardrails: install/doctor/scan → report.md + patches
Real-time surveillance dashboard for Claude Code Agent Teams — monitor agents, tasks, and messages live
CLI tool that monitors LLM rate limit utilization across Claude, Codex, and CLIProxyAPI — see how much of your paid quota you're actually using
Official SDK for Rulecatch — AI coding analytics, monitoring, and rule enforcement for Claude Code, Cursor, and AI coding assistants
Production-ready Laravel package for AI usage metering, token tracking, quota management, and Stripe billing integration. Supports OpenAI, Anthropic, and custom providers.
See which files burn your LLM tokens — fix them automatically. CI budget gate + MCP server + adapters for Claude, GPT-4o, Gemini, Ollama.
Local-first spend & cache-efficiency dashboard for Claude Code. Reads ~/.claude/projects, tells you exactly what's costing you. No telemetry, no API key.
Deterministic LLM testing for CI. Record→Replay + policy-as-code to block risky merges (PII, schema drift, regressions, budget creep).
Terminal UI for monitoring Claude Code token usage and costs
pytest for LLM apps - Test for grounding failures, prompt injection, safety violations, and regressions
The AI engineering platform for AI teams. Observability, evaluation, and prompt management for LLMs and AI agents. OpenTelemetry native.
🔍 Track and analyze Claude Code usage with rate limit awareness. Parse JSONL logs, calculate costs, and get warnings before hitting rate limits. CLI tool with daily/weekly breakdowns.
R Package: Pairwise Comparison Tools for LLM-Based Writing Evaluation
The Multi-LLM Benchmarking Tool