Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 15 of 57

AeroGraph★ 12

An open-source visual debugger for multi-agent AI, featuring local replay, loop detection, and trace branching.

cc-probeline★ 12

See where it leaks, stop paying for it — a live Claude Code status line that prices every turn, your subagents, cache rebuilds, plus limits, context and git.

open-llm-leaderboard-renamer★ 12

gradio region:us

open-llm-leaderboard-viz★ 12

streamlit region:us

kaz-llm-lb★ 12

gradio leaderboard region:us

bias-test-gpt-pairs★ 12

gradio region:us

LLMZSZL_Leaderboard★ 12

gradio region:us

Open-Persian-LLM-Leaderboard★ 12

gradio region:us

LLM-KV-cache-calculator★ 12

gradio region:us

EvalSurfer★ 12

Skill-first, agent-native evaluation protocol for AI apps

LLMPeek★ 12

Local devtools for LLM apps.

agentlens★ 12

Agentic AI monitoring and observability — OTEL traces and local log files from Copilot, Claude Code, Codex, and OpenCode. Token usage, cost, summaries, and efficiency insights. VS Code extension, npx, and Docker.

agentspy★ 12

"Educational tool for inspecting Claude Code ↔ LLM communication via hooks and proxy

claude-usage★ 12

Native macOS menu bar app showing Claude Code usage (session / weekly / Sonnet / credits) — no backend, notarized, brew install

llm-systems-manager★ 12

LLM Systems Manager — real-time observability and control for LLM systems

dsh-hud★ 12

Codex/Claude Code-style live session telemetry for DeepSeek Harness — model, context, tokens, timing, jobs, and subagents at a glance.

catchbench★ 12

CatchBench: a benchmark for auditing agent failures across the full lifecycle.

lightrace★ 12

A lightweight tracing platform for agent development

tau-emotion-bench★ 12

τ-Emotion-Bench: Exposing the Interaction Fragility of LLM Agents through Real-World User Emotions

Act-onomy★ 12

Actonomy: A taxonomy for describing and analyzing agent behavior. Gao et al., NeurIPS 2026 — How to Interpret Agent Behavior. Official codebook, datasets, and trace analysis tools (ACT*ONOMY / Act-onomy).

harness-opt★ 12

HOP — Harness Optimization Platform for continuously evaluating, qualifying, optimizing, and distributing local coding-agent configurations.

llm-reasoning-benchmarks★ 12

static region:us

chess-reasoning-llm★ 12

gradio mcp-server region:us

llm-latency-tracker★ 12

Regional AI API latency and uptime measurements. Open datasets, JSON API and MCP access; edge TTFB and inference TTFT reported separately.

token-usage-dashboard★ 12

📊 Beautiful dashboard for historical token usage for Claude, Codex and OpenCode

aerus-usage-hud★ 12

An AI Usage HUD for both Codex and Claud and Local Ollama in one place.

log10js★ 11

JavaScript client library for managing your LLM data in one place

presentation-2025-09-17-posit-conf-otel★ 11
HaystackCraft★ 11

Haystack Engineering: Context Engineering for Heterogeneous and Agentic Long-Context Evaluation

openclaw-monitor★ 11

🟢 Real-time 3D dashboard for OpenClaw AI agents — each agent is a continent, each session is a minion. Physics, live chat, thinking bubbles.

chub★ 11

Agent-agnostic context, tracking, and cost analytics for AI-assisted development. Curated docs · Session tracking · Cost analytics · Team knowledge — all git-native

zevals★ 11

Simple, practical AI agent testing in TypeScript

capsule★ 11

Capsule is an interactive inspector that turns session logs into a browsable and explorable timeline.

.github★ 11

Software Engineering Arena is an open-source initiative to transparently evaluate and track AI assistants across real-world software engineering tasks.

SWE-Wiki★ 11

Track Al coding assistants by wikis

SWE-Team★ 11

Track Al coding assistants by team management

SWE-Release★ 11

Track Al coding assistants by releases

claude-mission-control★ 11

Real-time command center for Claude Code agents. Monitor all agents from one dashboard, send instructions, track costs, detect stuck/looping agents, 7-layer security radar. Works from your phone via QR code.

claude-token-tracker★ 11

Token usage dashboard for Claude Code — tracks costs, sessions, models, and tools with real-time updates

Clawdmeter-Windows★ 11

Standalone Windows desktop dashboard for Claude Code usage — session & weekly limits with a reactive Clawd mascot. Software-only port of Clawdmeter, no ESP32 hardware.

italian_open_llm_leaderboard★ 11

streamlit region:us

evalita_llm_leaderboard★ 11

gradio region:us

logbook-judge★ 11

docker region:us

skyportalai★ 11

SkyPortal SDK & CLI

MT-InfoSeek★ 11

Do LLMs Know What to Ask and When? Evaluating Multi-Turn Information Seeking

toolreceipt★ 11

Catch LLM agents claiming actions they never took. Ledger + claim extraction + receipt for LangGraph.

webdevbench★ 10

A benchmark for evaluating AI coding platforms as on real-world web application development

ctxcraft★ 10

Evaluate and optimize your AI agent context. Save tokens, save money.

trace-taxi★ 10

Trace Taxi Trace Viewer

agent-cow★ 10

Agent Cow: the philosophical watcher of your agents across multiple machines.

MLflow-crash-course★ 10

A meticulously curated collection of hands-on Jupyter notebooks, designed to illuminate the comprehensive application of MLflow across a spectrum ranging from foundational Machine Learning principles to pioneering Generative AI paradigms.

simple-usage-monitor★ 10

CLI Tool - real-time token and cost estimator that lives in your terminal for Claude Code

helm★ 10

Helm Charts to deploy and manage OpenLIT in Kubernetes

waffie★ 10

LLM API automated testing tool

fiorino-ai★ 10

The open-source solution that helps SaaS owners track and optimize their AI costs.

LLMeval★ 10

Evaluate and compare large language models (LLMs) for chatbot applications, using various LLMs as evaluators, and manage prompt templates and binary preferences.

gpt-tokenator★ 10

GPT 3 tokens counter

GuessArena★ 10

[ACL 2025] GuessArena: Guess Who I Am? A Self-Adaptive Framework for Evaluating LLMs in Domain-Specific Knowledge and Reasoning

employee-teams-basic-bot★ 10

An M365 agentic app built on top of Teams AI library to demonstrate open-source Monocle instrumentation and Okahu Cloud observability for AI-enabled automated experiences.

ccusage-web★ 10

Web dashboard for monitoring Claude Code usage and costs. Built with Next.js, ECharts, and SQLite.

koog-spring-boot-assistant★ 10

Kotlin + SpringBoot + Koog + Promptfoo example

mirage-bench★ 10

Repository for Multililngual Generation, RAG evaluations, and surrogate judge training for Arena RAG leaderboard (NAACL'25)

website★ 10

Open Source OpenTelemetry-native Observability tool for GenAI and LLMs designed to provide developers with complete capabilities for monitoring, analysing, and optimising LLM applications.

llm-output-stability-gate★ 10

Pre-execution reliability gate using UQLM for LLM output stability

ccost★ 10

Accurate Claude API usage tracking and cost analysis tool. Features intelligent deduplication, multi-currency support, real-time monitoring, and comprehensive project analysis.

Lumina★ 10

A lightweight observability platform for LLM applications. Track costs, latency, and quality across your AI systems with minimal overhead.

LOGICOM★ 10
agentsonar★ 10

Coordination intelligence for any agent system

trace-wave★ 10

Datadog for AI Agents — Distributed tracing & observability for multi-agent systems. Trace every LLM call, tool use, and agent decision with 2 lines of code. Beautiful TUI + Web dashboard. Zero-config instrumentation for OpenAI/Anthropic/LangChain.

superopt★ 10

Agentic Environment Optimization for Autonomous AI Agents

local-agent-viewer★ 10

Your local long-term memory for AI agent interactions. Parse, classify, search, and visualize every interaction with Claude Code, Codex CLI, Claude Desktop, and ChatGPT.

Sub2APIStatusBar★ 10

macOS menu bar monitor for Sub2API user usage

spring-boot-ai★ 10

🍀 🤖 Spring Boot AI - Evaluation Testing

unworldly★ 10

The flight recorder for AI coding agents. Tamper-evident, verifiable audit trails of file changes and shell commands, with PHI pattern detection and agent identity.

agentlog★ 10

AgentLog is a lightweight event bus for AI agents built on append-only JSONL logs. Agents publish and subscribe to events via HTTP/SSE, enabling simple, replayable, and debuggable multi-agent workflows.

mcp-server★ 10

Stop shipping agents on vibes. Score every agent output for quality, safety, and cost.

AILens★ 10

Full-stack observability for AI — starting with end-to-end RL post-training, expanding to Agent and inference engine monitoring.

mocktopus★ 10

🐙 Multi-armed mocks for LLM apps - Drop-in replacement for OpenAI/Anthropic APIs for deterministic testing

token-saver★ 10

A local-first desktop app that finds, reduces, and verifies AI-agent token waste without sacrificing task quality.

ai-benchmarks-alternativos★ 10

Benchmark abierto en español de 170 modelos de IA (118 con 20+ runs, 69 rankeados, juez Phi-4 independiente). Calidad, costo, velocidad, long-context y fuga de credenciales como dimensiones separadas. Alternativas a Claude, GPT y Gemini para agentes n8n/Hermes. Calculadora interactiva con tus propios pesos.

llm-api-bench★ 10

Self-hosted LLM API benchmark, monitoring & playground. Compare latency, TTFT, throughput across OpenAI, Anthropic, Gemini & any OpenAI-compatible endpoint. Deploy with one command via Docker. | 自托管 LLM API 性能测试、监控与调试平台,一键 Docker 部署,支持多家服务商对比。

llm-usage-exporter★ 10

Self-hosted Prometheus exporter for LLM usage, token, request, and USD cost telemetry across OpenAI, Azure OpenAI, Anthropic Claude, Google Gemini, and AWS Bedrock.

Token-Dashboard★ 10

Local desktop dashboard for Claude Code. Reads your JSONL transcripts and surfaces per-prompt cost, tool heatmaps, subagent attribution, cache analytics, and a rule-based tips engine. Rust + Tauri, fully offline, MIT.

claude-code-cost★ 10

Cost dashboard for Claude Code usage

nemotron-opus-elicitation★ 10

29 controlled experiments on Nemotron 3 Ultra: what prompting can and cannot do to a frontier model. Voice 4/8 to 7/8, hidden bugs 1/5 to 5/5 — blind dual-judge graded, no fine-tuning.

CostHQ★ 10

Track what your AI agents cost including files, commits, tokens, budgets. CLI , Dashboard+ API. Works with OpenClaw, Claude Code, and any agent framework.

Build26-BRK241-from-prototype-to-production-build-and-run-agents-at-scale★ 10

Engineering patterns for taking AI agents to production — deployment, tools, memory, long-running work, human oversight, and observability at scale. From Microsoft Build 2026.

ai-workflow-benchmark★ 10

Benchmark harness measuring AI coding tool+workflow performance, not just model capability. 100 tasks, sigmoid scoring, 12 capability dimensions, gap analysis.

dspy-research★ 10

Experiments with DSPy — declarative, trainable LLM pipelines. Notebooks and demos for evaluating, composing, and optimizing LLM workflows (code-gen, company valuation, market analysis).

GroundEval★ 10

Observe an agent run and GroundEval drafts the policy and diagram for you, no hand-written policy required, then scores what it checked, what it skipped, and what it wasn't allowed to touch.

Tokensense★ 10

Open-source LLM cost observability and smart routing that runs in-process. No proxy, no telemetry, zero added latency.

Pacer★ 10

Native macOS app for tracking Claude Code usage — tokens, cost, rate-limit pacing, per-project breakdowns. SwiftUI + SwiftData.

pmstack★ 10

Gstack for PMs. Eval Studio that you can plug-n-play anywhere.

MAS-PromptBench★ 10

A reproducible benchmark for studying when prompt optimization improves multi-agent LLM systems.

verel★ 10

Verified agents — nothing is "done" until a grader returns a verdict. Eyes (AgentVision) + verdict bus + compounding memory + a fleet + agent-built tooling + agent-run CI/CD.

keras-chatbot-arena★ 10

gradio region:us

pinocchio-ita-leaderboard★ 10

gradio region:us

voight-vercel-ai★ 10

OpenTelemetry SpanExporter for the Vercel AI SDK → Voight. Drop-in observability

tku★ 10

Usage, cost and account toolkit for Claude Code, Codex and other AI coding agents

sample-GEDD★ 10

Find what your AI agent gets wrong — before you have a rubric. Qualitative eval for PMs.

More in Agent frameworks & infrastructure