Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 15 of 57
An open-source visual debugger for multi-agent AI, featuring local replay, loop detection, and trace branching.
See where it leaks, stop paying for it — a live Claude Code status line that prices every turn, your subagents, cache rebuilds, plus limits, context and git.
gradio region:us
streamlit region:us
gradio leaderboard region:us
gradio region:us
gradio region:us
gradio region:us
gradio region:us
Skill-first, agent-native evaluation protocol for AI apps
Local devtools for LLM apps.
Agentic AI monitoring and observability — OTEL traces and local log files from Copilot, Claude Code, Codex, and OpenCode. Token usage, cost, summaries, and efficiency insights. VS Code extension, npx, and Docker.
"Educational tool for inspecting Claude Code ↔ LLM communication via hooks and proxy
Native macOS menu bar app showing Claude Code usage (session / weekly / Sonnet / credits) — no backend, notarized, brew install
LLM Systems Manager — real-time observability and control for LLM systems
Codex/Claude Code-style live session telemetry for DeepSeek Harness — model, context, tokens, timing, jobs, and subagents at a glance.
CatchBench: a benchmark for auditing agent failures across the full lifecycle.
A lightweight tracing platform for agent development
τ-Emotion-Bench: Exposing the Interaction Fragility of LLM Agents through Real-World User Emotions
Actonomy: A taxonomy for describing and analyzing agent behavior. Gao et al., NeurIPS 2026 — How to Interpret Agent Behavior. Official codebook, datasets, and trace analysis tools (ACT*ONOMY / Act-onomy).
HOP — Harness Optimization Platform for continuously evaluating, qualifying, optimizing, and distributing local coding-agent configurations.
static region:us
gradio mcp-server region:us
Regional AI API latency and uptime measurements. Open datasets, JSON API and MCP access; edge TTFB and inference TTFT reported separately.
📊 Beautiful dashboard for historical token usage for Claude, Codex and OpenCode
An AI Usage HUD for both Codex and Claud and Local Ollama in one place.
JavaScript client library for managing your LLM data in one place
Haystack Engineering: Context Engineering for Heterogeneous and Agentic Long-Context Evaluation
🟢 Real-time 3D dashboard for OpenClaw AI agents — each agent is a continent, each session is a minion. Physics, live chat, thinking bubbles.
Agent-agnostic context, tracking, and cost analytics for AI-assisted development. Curated docs · Session tracking · Cost analytics · Team knowledge — all git-native
Simple, practical AI agent testing in TypeScript
Capsule is an interactive inspector that turns session logs into a browsable and explorable timeline.
Software Engineering Arena is an open-source initiative to transparently evaluate and track AI assistants across real-world software engineering tasks.
Track Al coding assistants by wikis
Track Al coding assistants by team management
Track Al coding assistants by releases
Real-time command center for Claude Code agents. Monitor all agents from one dashboard, send instructions, track costs, detect stuck/looping agents, 7-layer security radar. Works from your phone via QR code.
Token usage dashboard for Claude Code — tracks costs, sessions, models, and tools with real-time updates
Standalone Windows desktop dashboard for Claude Code usage — session & weekly limits with a reactive Clawd mascot. Software-only port of Clawdmeter, no ESP32 hardware.
streamlit region:us
gradio region:us
docker region:us
SkyPortal SDK & CLI
Do LLMs Know What to Ask and When? Evaluating Multi-Turn Information Seeking
Catch LLM agents claiming actions they never took. Ledger + claim extraction + receipt for LangGraph.
A benchmark for evaluating AI coding platforms as on real-world web application development
Evaluate and optimize your AI agent context. Save tokens, save money.
Trace Taxi Trace Viewer
Agent Cow: the philosophical watcher of your agents across multiple machines.
A meticulously curated collection of hands-on Jupyter notebooks, designed to illuminate the comprehensive application of MLflow across a spectrum ranging from foundational Machine Learning principles to pioneering Generative AI paradigms.
CLI Tool - real-time token and cost estimator that lives in your terminal for Claude Code
Helm Charts to deploy and manage OpenLIT in Kubernetes
LLM API automated testing tool
The open-source solution that helps SaaS owners track and optimize their AI costs.
Evaluate and compare large language models (LLMs) for chatbot applications, using various LLMs as evaluators, and manage prompt templates and binary preferences.
GPT 3 tokens counter
[ACL 2025] GuessArena: Guess Who I Am? A Self-Adaptive Framework for Evaluating LLMs in Domain-Specific Knowledge and Reasoning
An M365 agentic app built on top of Teams AI library to demonstrate open-source Monocle instrumentation and Okahu Cloud observability for AI-enabled automated experiences.
Web dashboard for monitoring Claude Code usage and costs. Built with Next.js, ECharts, and SQLite.
Kotlin + SpringBoot + Koog + Promptfoo example
Repository for Multililngual Generation, RAG evaluations, and surrogate judge training for Arena RAG leaderboard (NAACL'25)
Open Source OpenTelemetry-native Observability tool for GenAI and LLMs designed to provide developers with complete capabilities for monitoring, analysing, and optimising LLM applications.
Pre-execution reliability gate using UQLM for LLM output stability
Accurate Claude API usage tracking and cost analysis tool. Features intelligent deduplication, multi-currency support, real-time monitoring, and comprehensive project analysis.
A lightweight observability platform for LLM applications. Track costs, latency, and quality across your AI systems with minimal overhead.
Coordination intelligence for any agent system
Datadog for AI Agents — Distributed tracing & observability for multi-agent systems. Trace every LLM call, tool use, and agent decision with 2 lines of code. Beautiful TUI + Web dashboard. Zero-config instrumentation for OpenAI/Anthropic/LangChain.
Agentic Environment Optimization for Autonomous AI Agents
Your local long-term memory for AI agent interactions. Parse, classify, search, and visualize every interaction with Claude Code, Codex CLI, Claude Desktop, and ChatGPT.
macOS menu bar monitor for Sub2API user usage
🍀 🤖 Spring Boot AI - Evaluation Testing
The flight recorder for AI coding agents. Tamper-evident, verifiable audit trails of file changes and shell commands, with PHI pattern detection and agent identity.
AgentLog is a lightweight event bus for AI agents built on append-only JSONL logs. Agents publish and subscribe to events via HTTP/SSE, enabling simple, replayable, and debuggable multi-agent workflows.
Stop shipping agents on vibes. Score every agent output for quality, safety, and cost.
Full-stack observability for AI — starting with end-to-end RL post-training, expanding to Agent and inference engine monitoring.
🐙 Multi-armed mocks for LLM apps - Drop-in replacement for OpenAI/Anthropic APIs for deterministic testing
A local-first desktop app that finds, reduces, and verifies AI-agent token waste without sacrificing task quality.
Benchmark abierto en español de 170 modelos de IA (118 con 20+ runs, 69 rankeados, juez Phi-4 independiente). Calidad, costo, velocidad, long-context y fuga de credenciales como dimensiones separadas. Alternativas a Claude, GPT y Gemini para agentes n8n/Hermes. Calculadora interactiva con tus propios pesos.
Self-hosted LLM API benchmark, monitoring & playground. Compare latency, TTFT, throughput across OpenAI, Anthropic, Gemini & any OpenAI-compatible endpoint. Deploy with one command via Docker. | 自托管 LLM API 性能测试、监控与调试平台,一键 Docker 部署,支持多家服务商对比。
Self-hosted Prometheus exporter for LLM usage, token, request, and USD cost telemetry across OpenAI, Azure OpenAI, Anthropic Claude, Google Gemini, and AWS Bedrock.
Local desktop dashboard for Claude Code. Reads your JSONL transcripts and surfaces per-prompt cost, tool heatmaps, subagent attribution, cache analytics, and a rule-based tips engine. Rust + Tauri, fully offline, MIT.
Cost dashboard for Claude Code usage
29 controlled experiments on Nemotron 3 Ultra: what prompting can and cannot do to a frontier model. Voice 4/8 to 7/8, hidden bugs 1/5 to 5/5 — blind dual-judge graded, no fine-tuning.
Track what your AI agents cost including files, commits, tokens, budgets. CLI , Dashboard+ API. Works with OpenClaw, Claude Code, and any agent framework.
Engineering patterns for taking AI agents to production — deployment, tools, memory, long-running work, human oversight, and observability at scale. From Microsoft Build 2026.
Benchmark harness measuring AI coding tool+workflow performance, not just model capability. 100 tasks, sigmoid scoring, 12 capability dimensions, gap analysis.
Experiments with DSPy — declarative, trainable LLM pipelines. Notebooks and demos for evaluating, composing, and optimizing LLM workflows (code-gen, company valuation, market analysis).
Observe an agent run and GroundEval drafts the policy and diagram for you, no hand-written policy required, then scores what it checked, what it skipped, and what it wasn't allowed to touch.
Open-source LLM cost observability and smart routing that runs in-process. No proxy, no telemetry, zero added latency.
Native macOS app for tracking Claude Code usage — tokens, cost, rate-limit pacing, per-project breakdowns. SwiftUI + SwiftData.
Gstack for PMs. Eval Studio that you can plug-n-play anywhere.
A reproducible benchmark for studying when prompt optimization improves multi-agent LLM systems.
Verified agents — nothing is "done" until a grader returns a verdict. Eyes (AgentVision) + verdict bus + compounding memory + a fleet + agent-built tooling + agent-run CI/CD.
gradio region:us
gradio region:us
OpenTelemetry SpanExporter for the Vercel AI SDK → Voight. Drop-in observability
Usage, cost and account toolkit for Claude Code, Codex and other AI coding agents
Find what your AI agent gets wrong — before you have a rubric. Qualitative eval for PMs.