Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 18 of 57
GenPark AI Agent Skill - Post-processing confidence calibration using temperature scaling, expected calibration error (ECE), and Brier score evaluation.
GenPark AI Agent Skill - Decomposes total prediction uncertainty into epistemic (model knowledge void) vs aleatoric (inherent task ambiguity) components via Shannon mutual information.
A benchmark for evaluating LLM agents as sequential hyperparameter optimizers across 30 executable machine learning tasks
In-flight circuit breaker & loop recovery for LangGraph, LangChain, OpenAI & Anthropic agents. Catches reasoning loops, prevents runaway LLM costs with budget hard caps, and auto-injects Reflect & Pivot prompts.
Token Manager:Windows 本地 AI Token 用量、余额、成本与 API 代理监控工具
AI Agent Evaluation Platform - automated evaluation, multi-layer attribution analysis, data governance, Langfuse integration
τ-Rec: verifiable benchmarks for agentic conversational recommenders. The original movies benchmark (ACM RecSys 2026 Resource Track) and τ-Rec-Books, a controlled books benchmark on Wikidata (ECIR 2027 submission).
Mana Hub — self-hosted operations dashboard for your own AI agents. Next.js 16 + Supabase + Claude Code. Risk approvals, cron requests, web-push, knowledge base, file-drop bridge for tmux-hosted agents. Built for the 'Masterclass: Dein AI Agent Team' course.
A Scriptable widget for Codex quota and token usage.
Evidence-first monitoring for AI model releases, catalog changes, arenas, packages, docs, incidents, and platform signals.
Rage4j is a java library thats helps evaluate LLM's based on scientifically grounded metrics
Evidence-first accuracy hygiene for Claude.
DistDebug-Bench: Can LLM Agents Diagnose Bugs in Distributed Systems? (NeurIPS 2026 Evaluations & Datasets)
Deterministic runtime for agent evaluation
LangSmith tracing plugin for OpenClaw. Automatic observability for agent turns, tool calls, and LLM invocations.
Mission Control for your AI Agents
Semantic side-effect tracking for AI agents.
A living world where agents exist as participants alongside NPCs, internal actors, real service APIs, budgets, policies, and consequences.
Local-only observability for AI agents on your machine. One timeline across coding and non-coding agents.
Real-time observability for Claude Code agents. Track conversations, tool calls, and token usage across all sessions - zero config.
Sub-second RAG regression testing. Define golden questions, detect lost chunks in CI. pytest for your RAG pipeline.
Streamlit-based chatbot leveraging Ollama via LangChain and PostHog-LLM for advanced logging and monitoring
RuCa Benchmark (pronounced "roo-ka") - Russian Tool Calling Benchmark for LLM
Cloud Native Observability and Policy Engine for LLM Applications
A client for logging bot conversations in the Botmeter analytics platform
Native macOS menu-bar app that shows Claude Code and Codex usage windows at a glance, across multiple accounts.
Unit tests for AI prompts. Catch regressions before your users do.
HumanStudy-Bench: Community Edition — Standardized human study replays for AI agent evaluation
A YAML-config-driven CLI tool for evaluating AI agents
Multi-tier benchmark: Cultural grounding + Triad Engine eliminates LLM hallucination across Claude 4.6, GPT-5.2, Mistral 7B, Gemini 2.5 Pro. Raw 15-58% → 95-100% accuracy on 222 adversarial QA pairs (Ancient Rome 110 CE). Novel topological paradox detection (F1=0.939, zero-shot). Model-agnostic, in production.
🌍 Leaderboard Repository for "AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agent", ACL2024
A benchmark for agentic tool creation.
Samples to demonstrate how Monocle works for agentic applications
Desktop widget for tracking OpenAI, Anthropic, and OpenRouter usage, quotas, and rate limits.
See what each AI agent, client and job costs, in dollars at list price, and give any job a spend ceiling: preflight answers approved: false past it, and your code decides. An SDK and an HTTP API, not a proxy.
Application for comparing responses from different Large Language Models (LLMs) side-by-side
Zero-config LLM call inspector. See every API call your AI app makes.
Agent Reliability Engineering: applying SRE principles to AI agent systems. Evals, imp@k metrics, self-improvement, config versioning, transfer experiments.
Chrome extension showing Claude Pro/Max usage limits (Windows/Linux/macOS)
Hermes Agent operations cockpit for Kanban: health, assignment, progress, logs, output, and run history.
Root cause analysis for AI agents. Detects agent loops, retry storms, and optimization opportunities in LangSmith, Langfuse, Arize Phoenix, and OpenTelemetry traces.
Watch your Claude Code subagents work — a live office for every conversation.
btop for your AI coding agents — a single-binary local TUI showing live token & dollar cost for Claude Code, with a budget bar that turns red before your bill does. No signup, no network.
Reference AI platform for architects — FastAPI gateway integrating RAG, Agents, and MLflow with audit logging, FinOps metrics, RBAC, and Grafana observability.
🐾🔍 The tool-call detective for small models on Apple Silicon — attributes every tool-calling failure to one of four causes (template bug / parser gap / model format / model decision). MLX-native, reproducible, bootstrap CIs on every metric.
Real-time Claude Code usage in your macOS menu bar — burn-rate ETA · 7-day sparkline · custom alerts · CSV/JSON export · 4 languages (EN/KO/JA/ZH)
Open-source observability and monitoring platform for AI agents. Real-time tracing, metrics, cost tracking, and debugging for production AI agent systems.
AEP is an open observability protocol for AI agent systems - giving you structured event capture, real-time session tracing, and multi-agent workflow visibility in a single self-hosted deployment.
Live index of LLM evaluation tools and benchmarks, refreshed every 15 minutes from GitHub
docker region:us
gradio region:us
gradio leaderboard region:us
static region:us
macOS menu-bar app that watches your Claude & Codex usage and auto-switches accounts before you hit a rate limit.
A cute desktop pet with a live Claude token usage meter - session & weekly rate limits in a thought cloud above your Windows taskbar. Free, open source, one-press install.
Detect prompt regressions before they reach production — per-category accuracy scoring, deterministic validation, and False Improvement detection. Pure Python, zero dependencies.
Local prompt-cache economics for LLM agents: split cache reads/writes from input spend, reconcile dollars to invoices, and surface concrete TTL/marker fixes.
Native Windows tray monitor for Claude Code usage — shows your rate-limit %, projects when you'll run out, and turns the icon red before you hit the limit. Plus a local 24h usage breakdown.
A Observability platform for Rag application, supports built in langchain and llamaindex integration
Local Tokenizer/Counter
static region:us
Local-first flight recorder for AI agents: every LLM call, tool call, cost, and loop captured by a transparent proxy. Zero code changes.
Local-first desktop app for building LLM test suites and running them against many local or cloud models at once
Windows system-tray monitor for Claude usage limits - live session/weekly meters, usage graph, alerts. Unofficial.
Self-Consistency majority voting engine aggregating multiple stochastic reasoning paths to extract high-confidence consensus solutions.
Production-grade genpark-hdr-histogram-percentile-estimator-skill skill for AI agents
Production-grade genpark-w3c-trace-context-span-propagator-skill skill for AI agents
Subscription window keeper that keeps AI coding plan windows active, locally or via a remote server.
Guide to Claude API usage: getting Console access, reading cost and usage reports, and keeping API spend in view.
Per-turn token and cost monitoring for Codex and Claude Code.
Agent 测试转型手册:资深自动化测试工程师的 AI QE 学习路径 + 练手代码
Build self-improving AI agent harnesses with portable harness units, artifact-aware testing, trace-backed diagnosis, and evidence-gated promotion.
HEAMS: Holistic Evaluation of Agent Memory Systems. An open framework for evaluating memory systems for AI agents on performance and privacy.
Universal Autonomous Agent Runtime Self-Healing & Diagnostics Engine. Detects configuration drift, corrupted execution state, and missing runtime resources, executing automated root-cause classification and snapshot rollback.
Production-Agentic-AI-Eval-Harness
Accountability infrastructure for long-running AI agents. Self-hosted MCP server: identity, claims and evidence, review, outcomes, and reconstruction across agent runtimes.
Codex 管家 · 多智能体 AI 会话管家:Token 用量看板 + 问答追踪 + SVG 流程画布 / 分布工作区 / 神经网络文字树图(单文件编辑器 + 本地服务,七家智能体适配)| Multi-agent AI session butler: usage dashboards, Q&A tracking, visual flow canvas, distributed workspace & neural text-treemap
macOS menu bar port of Burn Bar: AI token burn and plan pacing for Claude, Codex and Grok
Reproducible AI reliability workbench: seeded faults, trace-complete replay, paired recovery experiments, uncertainty-aware policy selection, and CI-audited evidence.
gradio llm-evaluation rlhf hallucination-detection chain-of-thought bayesian benchmark region:us
Forecasts LLM token consumption trajectories, enforces hard multi-agent cost budgets, and automatically truncates low-priority context windows | GenPark: https://www.genpark.ai | LinkedIn: https://www.llinkedin.com/company/92955550
Agent Reliability Toolkit for LLMs - Test, evaluate, stress-test, and red-team your AI applications with scenario-based testing, multiple evaluators, and multi-provider support.
Model Finder: find the cheapest AI model that passes your own tests. MCP server + Agent Skills for model audits, evals, cheapest-first routing and local model sizing.
Google Summer of Code 2025 Project: A comprehensive benchmark for evaluating Large Language Models (LLMs) across the five classic AI agent architectures
AI Assistant for FMBench (foundation model benchmarking tool)
Statistically sound A/B tests for prompts and models: paired tests, confidence intervals, sample-size planning and a CI regression gate. Type-safe prompts with Pydantic.
Keep you informed about your OpenAI/Anthropic API usage costs
HumanEval 模型评估工具
EntRAG - Enterprise RAG Benchmark
A framework for evaluating various customization techniques for foundation models (FMs) using your own dataset. This includes approaches like RAG, fine-tuning, and a hybrid method that combines both
LLM observability for React Native and Expo. Track token usage, cost, latency, and TTFT for Claude, GPT, and Gemini calls on device, with optional remote sync.
Open-source agent observability — session replay, metrics, and shareable snapshots for AI agent workflows
Python framework for evaluating LLM tool-calling behavior with comprehensive metrics on accuracy, efficiency, and correctness
Chrome DevTools for AI Agents — real-time observability, hallucination detection, session replay, and cost tracking. MIT licensed.
Research benchmark: conversational memory learning on a corrected LoCoMo dataset
A Python package for tracking and analyzing LLM usage across different models and applications. It is primarily designed as a library for integration into development process of LLM-based agentic workflow tooling, providing robust tracking capabilities.
Deterministic regression testing for AI agents