Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 18 of 57

genpark-temperature-scaling-confidence-calibrator-skill★ 8

GenPark AI Agent Skill - Post-processing confidence calibration using temperature scaling, expected calibration error (ECE), and Brier score evaluation.

genpark-agent-epistemic-aleatoric-uncertainty-decomposer-skill★ 8

GenPark AI Agent Skill - Decomposes total prediction uncertainty into epistemic (model knowledge void) vs aleatoric (inherent task ambiguity) components via Shannon mutual information.

AgentHPOBench★ 8

A benchmark for evaluating LLM agents as sequential hyperparameter optimizers across 30 executable machine learning tasks

LongGuard★ 8

In-flight circuit breaker & loop recovery for LangGraph, LangChain, OpenAI & Anthropic agents. Catches reasoning loops, prevents runaway LLM costs with budget hard caps, and auto-injects Reflect & Pivot prompts.

token-manager★ 8

Token Manager:Windows 本地 AI Token 用量、余额、成本与 API 代理监控工具

agentbench★ 8

AI Agent Evaluation Platform - automated evaluation, multi-layer attribution analysis, data governance, Langfuse integration

tau-rec★ 8

τ-Rec: verifiable benchmarks for agentic conversational recommenders. The original movies benchmark (ACM RecSys 2026 Resource Track) and τ-Rec-Books, a controlled books benchmark on Wikidata (ECIR 2027 submission).

mana-hub★ 8

Mana Hub — self-hosted operations dashboard for your own AI agents. Next.js 16 + Supabase + Claude Code. Risk approvals, cron requests, web-push, knowledge base, file-drop bridge for tmux-hosted agents. Built for the 'Masterclass: Dein AI Agent Team' course.

codex-usage-widget★ 8

A Scriptable widget for Codex quota and token usage.

signal-forge★ 8

Evidence-first monitoring for AI model releases, catalog changes, arenas, packages, docs, incidents, and platform signals.

quarkus-rage4j★ 8

Rage4j is a java library thats helps evaluate LLM's based on scientifically grounded metrics

llm-accuracy★ 8

Evidence-first accuracy hygiene for Claude.

distdebug-bench★ 8

DistDebug-Bench: Can LLM Agents Diagnose Bugs in Distributed Systems? (NeurIPS 2026 Evaluations & Datasets)

Tracecore★ 7

Deterministic runtime for agent evaluation

openclaw-langsmith★ 7

LangSmith tracing plugin for OpenClaw. Automatic observability for agent turns, tool calls, and LLM invocations.

remote-control-on-demand★ 7

Mission Control for your AI Agents

Trace-lit-SDK★ 7
effect-log★ 7

Semantic side-effect tracking for AI agents.

volnix★ 7

A living world where agents exist as participants alongside NPCs, internal actors, real service APIs, budgets, policies, and consequences.

agentwatch★ 7

Local-only observability for AI agents on your machine. One timeline across coding and non-coding agents.

Marionette★ 7

Real-time observability for Claude Code agents. Track conversations, tool calls, and token usage across all sessions - zero config.

LongProbe★ 7

Sub-second RAG regression testing. Define golden questions, detect lost chunks in CI. pytest for your RAG pipeline.

streamlit-chatbot-analytics★ 7

Streamlit-based chatbot leveraging Ollama via LangChain and PostHog-LLM for advanced logging and monitoring

RuCa★ 7

RuCa Benchmark (pronounced "roo-ka") - Russian Tool Calling Benchmark for LLM

Observicia★ 7

Cloud Native Observability and Policy Engine for LLM Applications

botmeter-logger★ 7

A client for logging bot conversations in the Botmeter analytics platform

LLimit★ 7

Native macOS menu-bar app that shows Claude Code and Codex usage windows at a glance, across multiple accounts.

prompt-ci★ 7

Unit tests for AI prompts. Catch regressions before your users do.

HumanStudy-Bench★ 7

HumanStudy-Bench: Community Edition — Standardized human study replays for AI agent evaluation

agent-eval★ 7

A YAML-config-driven CLI tool for evaluating AI agents

hallucination-elimination-benchmark★ 7

Multi-tier benchmark: Cultural grounding + Triad Engine eliminates LLM hallucination across Claude 4.6, GPT-5.2, Mistral 7B, Gemini 2.5 Pro. Raw 15-58% → 95-100% accuracy on 222 adversarial QA pairs (Ancient Rome 110 CE). Novel topological paradox detection (F1=0.939, zero-shot). Model-agnostic, in production.

appworld-leaderboard★ 7

🌍 Leaderboard Repository for "AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agent", ACL2024

ToolArena★ 7

A benchmark for agentic tool creation.

examples★ 7

Samples to demonstrate how Monocle works for agentic applications

PeekaUsage★ 7

Desktop widget for tracking OpenAI, Anthropic, and OpenRouter usage, quotas, and rate limits.

agentbill★ 7

See what each AI agent, client and job costs, in dollars at list price, and give any job a spend ceiling: preflight answers approved: false past it, and your code decides. An SDK and an HTTP API, not a proxy.

LLM-Diff-Tool★ 7

Application for comparing responses from different Large Language Models (LLMs) side-by-side

llm-spy★ 7

Zero-config LLM call inspector. See every API call your AI app makes.

agent-reliability-engineering★ 7

Agent Reliability Engineering: applying SRE principles to AI agent systems. Evals, imp@k metrics, self-improvement, config versioning, transfer experiments.

claude-usage-extension★ 7

Chrome extension showing Claude Pro/Max usage limits (Windows/Linux/macOS)

h-ops★ 7

Hermes Agent operations cockpit for Kanban: health, assignment, progress, logs, output, and run history.

sentinel★ 7

Root cause analysis for AI agents. Detects agent loops, retry storms, and optimization opportunities in LangSmith, Langfuse, Arize Phoenix, and OpenTelemetry traces.

claude-theater★ 7

Watch your Claude Code subagents work — a live office for every conversation.

toktop★ 7

btop for your AI coding agents — a single-binary local TUI showing live token & dollar cost for Claude Code, with a budget bar that turns red before your bill does. No signup, no network.

ai-architect★ 7

Reference AI platform for architects — FastAPI gateway integrating RAG, Agents, and MLflow with audit logging, FinOps metrics, RBAC, and Grafana observability.

toolhound★ 7

🐾🔍 The tool-call detective for small models on Apple Silicon — attributes every tool-calling failure to one of four causes (template bug / parser gap / model format / model decision). MLX-native, reproducible, bootstrap CIs on every metric.

ClaudeUsageWidget★ 7

Real-time Claude Code usage in your macOS menu bar — burn-rate ETA · 7-day sparkline · custom alerts · CSV/JSON export · 4 languages (EN/KO/JA/ZH)

agent-monitor★ 7

Open-source observability and monitoring platform for AI agents. Real-time tracing, metrics, cost tracking, and debugging for production AI agent systems.

agent-event-protocol★ 7

AEP is an open observability protocol for AI agent systems - giving you structured event capture, real-time session tracing, and multi-agent workflow visibility in a single self-hosted deployment.

llm-eval-tracker★ 7

Live index of LLM evaluation tools and benchmarks, refreshed every 15 minutes from GitHub

benchmarks★ 7

docker region:us

GenerationVisualizer★ 7

gradio region:us

open-ko-llm-leaderboard-old★ 7

gradio leaderboard region:us

agent-artifacts★ 7

static region:us

chewy★ 7

macOS menu-bar app that watches your Claude & Codex usage and auto-switches accounts before you hit a rate limit.

hachiko-usage★ 7

A cute desktop pet with a live Claude token usage meter - session & weekly rate limits in a thought cloud above your Windows taskbar. Free, open source, one-press install.

prompt-regression-suite★ 7

Detect prompt regressions before they reach production — per-category accuracy scoring, deterministic validation, and False Improvement detection. Pure Python, zero dependencies.

cacheeconomics★ 7

Local prompt-cache economics for LLM agents: split cache reads/writes from input spend, reconcile dollars to invoices, and surface concrete TTL/marker fixes.

claude-tray★ 7

Native Windows tray monitor for Claude Code usage — shows your rate-limit %, projects when you'll run out, and turns the icon red before you hit the limit. Plus a local 24h usage breakdown.

RagObserve★ 7

A Observability platform for Rag application, supports built in langchain and llamaindex integration

tcount★ 7

Local Tokenizer/Counter

AgentGen-Bench★ 7

static region:us

AgenticLedger★ 7

Local-first flight recorder for AI agents: every LLM call, tool call, cost, and loop captured by a transparent proxy. Zero code changes.

cannonade★ 7

Local-first desktop app for building LLM test suites and running them against many local or cloud models at once

claude-meter-for-windows★ 7

Windows system-tray monitor for Claude usage limits - live session/weekly meters, usage graph, alerts. Unofficial.

genpark-self-consistency-majority-voter-skill★ 7

Self-Consistency majority voting engine aggregating multiple stochastic reasoning paths to extract high-confidence consensus solutions.

genpark-hdr-histogram-percentile-estimator-skill★ 7

Production-grade genpark-hdr-histogram-percentile-estimator-skill skill for AI agents

genpark-w3c-trace-context-span-propagator-skill★ 7

Production-grade genpark-w3c-trace-context-span-propagator-skill skill for AI agents

awewarm★ 7

Subscription window keeper that keeps AI coding plan windows active, locally or via a remote server.

claude-api-usage★ 7

Guide to Claude API usage: getting Console access, reading cost and usage reports, and keeping API spend in view.

turnlens★ 7

Per-turn token and cost monitoring for Codex and Claude Code.

agent-testing-handbook★ 7

Agent 测试转型手册:资深自动化测试工程师的 AI QE 学习路径 + 练手代码

Harneloop★ 7

Build self-improving AI agent harnesses with portable harness units, artifact-aware testing, trace-backed diagnosis, and evidence-gated promotion.

heams★ 7

HEAMS: Holistic Evaluation of Agent Memory Systems. An open framework for evaluating memory systems for AI agents on performance and privacy.

genpark-agent-runtime-self-healing-diagnostics-skill★ 7

Universal Autonomous Agent Runtime Self-Healing & Diagnostics Engine. Detects configuration drift, corrupted execution state, and missing runtime resources, executing automated root-cause classification and snapshot rollback.

Agentic-AI-Eval-Harness★ 7

Production-Agentic-AI-Eval-Harness

unitares★ 7

Accountability infrastructure for long-running AI agents. Self-hosted MCP server: identity, claims and evidence, review, outcomes, and reconstruction across agent runtimes.

codex-manager★ 7

Codex 管家 · 多智能体 AI 会话管家:Token 用量看板 + 问答追踪 + SVG 流程画布 / 分布工作区 / 神经网络文字树图(单文件编辑器 + 本地服务,七家智能体适配)| Multi-agent AI session butler: usage dashboards, Q&A tracking, visual flow canvas, distributed workspace & neural text-treemap

cinderbar★ 7

macOS menu bar port of Burn Bar: AI token burn and plan pacing for Claude, Codex and Grok

faultline★ 7

Reproducible AI reliability workbench: seeded faults, trace-complete replay, paired recovery experiments, uncertainty-aware policy selection, and CI-audited evidence.

llm-reasoning-evaluator★ 7

gradio llm-evaluation rlhf hallucination-detection chain-of-thought bayesian benchmark region:us

genpark-token-cost-forecaster-and-context-budgeter-skill★ 7

Forecasts LLM token consumption trajectories, enforces hard multi-agent cost budgets, and automatically truncates low-priority context windows | GenPark: https://www.genpark.ai | LinkedIn: https://www.llinkedin.com/company/92955550

artemiskit★ 7

Agent Reliability Toolkit for LLMs - Test, evaluate, stress-test, and red-team your AI applications with scenario-based testing, multiple evaluators, and multi-provider support.

EvalHarness★ 7

Model Finder: find the cheapest AI model that passes your own tests. MCP server + Agent Skills for model audits, evals, cheapest-first routing and local model sizing.

AgenticEvals★ 6

Google Summer of Code 2025 Project: A comprehensive benchmark for evaluating Large Language Models (LLMs) across the five classic AI agent architectures

fmbench-assistant★ 6

AI Assistant for FMBench (foundation model benchmarking tool)

flowprompt★ 6

Statistically sound A/B tests for prompts and models: paired tests, confidence intervals, sample-size planning and a CI regression gate. Type-safe prompts with Pydantic.

sonoma-dusk-sky-alpha-evaluation★ 6
OpenAIAPIUsage★ 6

Keep you informed about your OpenAI/Anthropic API usage costs

human-eval-testbed★ 6

HumanEval 模型评估工具

EntRAG★ 6

EntRAG - Enterprise RAG Benchmark

tailoring-foundation-models-for-business-needs-guide-to-rag-fine-tuning-hybrid-a★ 6

A framework for evaluating various customization techniques for foundation models (FMs) using your own dataset. This includes approaches like RAG, fine-tuning, and a hybrid method that combines both

Benchmarking-SLMs-and-SRLMs-on-System-Log-Severity-Classification★ 6
react-native-llm-meter★ 6

LLM observability for React Native and Expo. Track token usage, cost, latency, and TTFT for Claude, GPT, and Gemini calls on device, with optional remote sync.

claw-insights★ 6

Open-source agent observability — session replay, metrics, and shareable snapshots for AI agent workflows

Toolscore★ 6

Python framework for evaluating LLM tool-calling behavior with comprehensive metrics on accuracy, efficiency, and correctness

agentlens★ 6

Chrome DevTools for AI Agents — real-time observability, hallucination detection, session replay, and cost tracking. MIT licensed.

roampal-labs★ 6

Research benchmark: conversational memory learning on a corrected LoCoMo dataset

llm-accounting★ 6

A Python package for tracking and analyzing LLM usage across different models and applications. It is primarily designed as a library for integration into development process of LLM-based agentic workflow tooling, providing robust tracking capabilities.

trajectly★ 6

Deterministic regression testing for AI agents

More in Agent frameworks & infrastructure