Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 20 of 57

noctrace★ 5

Chrome DevTools Network-tab-style waterfall visualizer for Claude Code agent workflows. Zero config, zero cloud.

claude-pulse★ 5

Real-time Claude AI usage monitor for Windows — hover the tray, see your session, weekly, and monthly limits.

GranSabio_LLM★ 5

Multi-Layer AI Quality Assurance for Content Generation. Multiple LLMs evaluate, score, and approve every output before delivery.

token-tally★ 5

An LLM Cost Calculator for all the major services

Instrumental-Choices★ 5

Benchmark for measuring instrumental-convergence behaviour in tool-using LLM agents

claude-meter★ 5

Real-time Claude AI subscription usage monitor for Windows — ultra-lightweight Rust system tray app (~3 MB RAM)

burnd★ 5

Find what's burning a hole in your AI coding budget. Local-first cost-control for Claude Code.

usage-widget★ 5

macOS desktop widget showing Claude Code usage as a contribution grid with real-time rate limit bars

liment★ 5

Convenient macOS app to display LLM usage limits in the tray.

tempo-for-claude★ 5

Tempo - macOS menu bar app that tracks Claude Code token and credit usage. Polls the Anthropic API via OAuth and syncs data to iCloud, which an iOS companion reads and forwards to Apple Watch via WatchConnectivity. Delivers haptic alerts on Apple Watch when a session ends. Built with SwiftUI across macOS, iOS, and watchOS targets.

fakellm★ 5

Mock OpenAI and Anthropic APIs locally for fast, deterministic tests.

Pulse-Claude-Code-Analytics★ 5

Local-first analytics for Claude Code, OpenAI Codex and OpenCode. Track tokens, costs, cache efficiency and account limits, with Discord Rich Presence. Built with Rust and Tauri.

agent-bench★ 5

Industrial-grade benchmarking engine for AI agents. Define test scenarios in YAML, run high-performance parallel evaluations, and generate premium glassmorphism reports. Supports LangGraph, CrewAI, AutoGen, and custom agent stacks.

langgraph-vs-crewai★ 5

A production-grade benchmark suite using Welch's T-Tests and Cohen's D to objectively compare AI agent frameworks (LangGraph, CrewAI, AutoGen).

ContextTax★ 5

Measure the context-window tax an MCP server charges your AI agent — the real token cost of its tool schemas + responses. Ground truth (Anthropic count_tokens) or keyless estimate. Cross-platform single-file CLI.

jilog★ 5

Pluggable session-log review and append-only event ledger

ground-control★ 5

Mission Control for cost-conscious OpenClaw enthusiasts. Lightweight CLI/TUI tools for agent cost monitoring, task management, and configuration.

agentdog★ 5

AgentDog helps developers inspect, test, score, and monitor AI agent runs locally.

skill-probe★ 5

Runtime-agnostic, co-loaded-aware auditor for AI agent skill libraries — measures which skill actually fires, with statistical confidence + a Bayesian fix-loop.

probe★ 5

AgentEval — probe your AI agents the way real users do: browser-driven, natural-language scenarios, AI-as-judge, multi-dimensional reports.

agent-anvil-leaderboard★ 5

Public Agent Anvil leaderboard submissions and generated index

AgentRoyale★ 5

Unit tests for AI agents that browse the web.

sentinel★ 5

Open-source observability and trust layer for AI agents: trace every step, score every output, catch hallucinations and runaway loops in real time. Self-hostable.

memvidbench★ 5

Benchmark tool for evaluating Memvid on the LoCoMo (Long-term Conversational Memory) benchmark.

agent-eval-harness★ 5

Evaluate AI agents with Unix-style pipeline commands. Schema-driven adapters for any CLI agent, trajectory capture, pass@k metrics, and multi-run comparison.

arena★ 5

OutplayArena is a platform to facilitate game-theoretic analyses between AI agents, and, optionally humans.

mcp-token-auditor★ 5

A high-performance, multi-agent observability engine designed for the Model Context Protocol (MCP). It provides a non-blocking, transparent proxy layer that implements deterministic token attribution, real-time context-window alerting, and heuristic-driven static analysis to optimize LLM metadata overhead at scale.

codex-rate-watcher★ 5

⚡ A macOS menu bar app for Codex power users: real-time quota visibility, multi-account relay, and all-device token cost syncing through iCloud. Pure Swift.

ai-coding-usage-tracker★ 5

Local-first AI coding usage dashboard for Codex, Claude Code, and Cursor — tokens, credits, USD estimates, native GUI, and WakaTime sync.

codex-insights★ 5

Codex CLI session analyzer - generates HTML usage reports with AI-powered suggestions

cdxusage★ 5

The fastest measured, optimized local Codex usage meter for tokens, sessions, and estimated OpenAI cost.

claude-monitor-lite★ 5

Lightweight macOS menu bar app to monitor your Claude account's real-time rate limits with auto-refresh and scheduled session renewal

cc-token★ 5

Claude Code usage dashboard in your macOS menu bar — costs, tokens, plan limits, trends, multi-machine sync

claude-pacer★ 5

A macOS menu bar app that paces your Claude usage — 5-hour & 7-day gauges plus scheduled pings that align fresh 5-hour windows to your core hours. Swift/SwiftUI. For Claude Pro/Max.

ragproof★ 5

Test harness for RAG pipelines: scores retrieval, groundedness, citation accuracy and prompt-injection resistance, and fails CI on regression.

traceguard-ai★ 5

Autonomous quality firewall for LLM agents - classifies failures, generates patches, writes evaluators, and runs shadow deployments. Built on LangSmith + LangGraph + Groq.

claude-code-usage-per-project★ 5

Per-project usage & cost tracking for Claude Code — status line + terminal reports

tokenwise★ 5

🪙 A local, zero-dependency token-cost profiler for Claude Code sessions. Finds what actually burned your tokens (big results that re-bill every turn, redundant reads, runaway subagents) and how to cut it.

claude-convo-analyzer★ 5

A local-only web app that analyzes your Claude Code conversation logs for per-conversation token usage and cost.

VS-Code-Claude-Usage★ 5

A simple status bar & tooltip showing your Claude Code usage metrics.

claude-meter★ 5

macOS menu bar app showing Claude usage & limits (session/weekly, Codex rate limits, API spend). Multi-account, JP/EN.

rlg★ 5

Near-lock-free structured logging for Rust. 10-crate workspace: MCP server (Claude/Cursor/mcp.run), OTLP HTTP+gRPC exporter, Tower middleware, WASM + WASI 0.2 component, eBPF-style enrichers, PII redaction, log analytics, test utilities, and a jq-style CLI. Sub-µs ingest via a 65k-slot ring buffer (LMAX Disruptor).

Persian-LLM-Leaderboard★ 5

streamlit region:us

open_llm_leaderboard★ 5

gradio region:us

Swahili_LLM_Leaderboard★ 5

streamlit region:us

k2-eval-gallery★ 5

streamlit region:us

llm-human-feedback-collector-chat-interface-kto★ 5

gradio region:us

LLM-Microscope★ 5

gradio region:us

Guide_Evaluation_LLM★ 5

streamlit region:us

41-llms-evaluated-locally-on-19-benchmarks★ 5

static region:us

konkur1404-llm-leaderboard★ 5

gradio region:us

jua-leaderboard★ 5

docker region:us

wc2026-llm-leaderboard★ 5

gradio region:us

LLM-Training-Time-and-Cost-Calculator★ 5

gradio region:us

Local-Open-LLM-Performance-Leaderboard★ 5

static region:us

41-llms-evaluated-locally-on-19-benchmarks★ 5

static region:us

agent-chatlens★ 5

🔍 A beautiful web viewer for AI agent session files. Browse Claude Code & OpenClaw conversations with chat-style UI, timeline visualization, and zero setup.

IssueBenchKit★ 5

Turn a real GitHub issue into a repeatable coding-agent benchmark task

agentlen★ 5

Local-first Claude Code and Codex session log viewer with search, replay, and usage analytics.

agentomaly★ 5

Behavioral anomaly detection and runtime observability for AI agents

harness-dashboard★ 5

One fuel gauge for every AI coding harness you actually use.

lattice★ 5

Interactive dashboard that visualizes agent workflows from project files

pnlc-agentclinic★ 5

This repository is based on a study that if the critic overcome the generators failures or they inherit them.

AgentGuard★ 5

Open-source CLI framework to regression-test AI agents with deterministic rules, provider execution and CI-ready reports.

ezpz-evals★ 5

Local-first eval harness for unstructured-document extraction — compare LLMs, OCR/IDP tools, and strategies (cascade/ensemble/verify) on the same cohort.

codex-token-dashboard★ 5

本地优先的 Codex Token 用量看板,支持官方 Plus、中转站、缓存命中率、脱敏快照和 Netlify 静态演示。

Tokens4Breakfast-daily★ 5

Track AI token usage, spend, subscriptions & rate-limit pressure across 8 providers (Claude Code, OpenAI, Cursor, Copilot, Gemini, DeepSeek, Mistral, OpenRouter) in your Mac menu bar. Local-first. A ccusage alternative with a GUI.

metarag-sdk★ 5

MetaRAG is an open-source engine that takes the guesswork out of RAG pipeline design. Instead of manually tuning chunking strategies, retrieval backends, and rerankers, MetaRAG benchmarks them against your own queries and routes each one to whichever configuration actually performs best on your data.

rag-evals★ 5

CI-friendly RAG evaluation: retrieval metrics, LLM-as-judge faithfulness, and a regression gate that fails your build when quality drops

verbatimeter★ 5

Minimalist, deterministic, lightweight, fast, and portable verification of AI-generated text: measure groundedness in a source as verbatim reuse (contiguous matching) or verbatim paraphrasing (LCS). Use as a decorator over generate(), a CLI or a CI gate. Useful for research and production purposes.

DataClawEval★ 5

DataClawEval: A Benchmark for Engineering Data Agents in Real Industrial Harness

claude-code-token-analyzer★ 5

A browser-based dashboard for measuring the impact of Claude Code optimization tools (Graphify, Claude-Mem, opusplan) using ccusage session data. No install required.

agent-replay★ 5

Watch a Claude agent's tool-calling loop — paste a trace, replay it step by step on a cinematic timeline. No key, no backend.

claude-prompt-meter★ 5

Per-prompt token & cost meter for Claude Code — status-bar meter, session dashboard with spend heatmap, automatic per-model pricing, daily budgets. Thai/English.

contextbenchmark★ 5

The open benchmark for measuring the reliability, reproducibility, and determinism of AI context systems. Transparent metrics, reproducible tests, vendor-neutral results. Context Trust Levels CTL 0-4.

Aether★ 5

High-conviction AI engineering workbench. Local-first, zero slop. Prefix-cache-safe routing, binary Plan/Act gates, and invisible AutoMemory.

tokentelemetry-hermes-plugin★ 5

TokenTelemetry launcher tab inside Hermes Dashboard — local observability for Hermes Agent and 9 coding agents.

stet-cli★ 5

Evaluate coding agents (Claude Code, Codex, Cursor) on your own repo's merged PRs. Installer and agent skill for the Stet CLI.

dgx-spark-bench★ 5

Real, reproducible LLM inference benchmarks on the NVIDIA DGX Spark — parallel agent sessions, tok/s and latency under load, with an interactive dashboard and one-command recipes

claude-plugin-cxdb-integration★ 5

Claude Code plugin that records every session as structured turns in cxdb via lifecycle hooks, enabling trajectory browsing, tool-call inspection, and token-usage observability.

trapstreet-skills★ 5

Agent skills for trapstreet.run — set up the tp CLI, build solutions against an eval task, and author new tasks, from plain language. Claude Code, Cursor, Codex and 70+ more.

awesome-agentic-usecases★ 5

Verified agentic AI evaluation labs, public-sector assurance tools, reproducible failures, real-model evidence, and provider-neutral test contracts.

Self-Evolving-Agents-Double-Ratchet★ 5

Reference implementation of Double Ratchet: co-evolving an inspectable evaluation metric with a lifecycle-managed skill library for self-improving LLM agents (arXiv:2607.12790)

QuotaBar★ 5

Free, open-source Codex usage tracker for the macOS menu bar: monitor 5-hour/weekly limits and reset times locally. 免费开源的 Codex 用量监控工具。

streamdeck-dashboard★ 5

OpenDeck plugin that shows local Codex task status on a Stream Deck key

Clew-by-Custos★ 5

Open source LLM observability for the waste axis - 4 deterministic detectors + LLM-as-Judge, waste-rate metric verified across 3 corpora / 16,864 sessions

edge-research★ 5

EdgeResearch — autonomously improve any named artifact (prompts, copy, code, system, perf) through measurable experiments. Methodology port of @karpathy's autoresearch.

langfuse-proxy★ 5

Lightweight Bun + Elysia proxy for OpenAI-compatible APIs with Langfuse telemetry

glint★ 5

Laravel-native LLM observability, traces, cost tracking, latency, errors, and alerts for your AI calls

llm-quota★ 5

Local-first live dashboard, CLI, and Windows widget for your subscription quotas plus local token spend. Silent updates, zero telemetry.

harness-lens★ 5

Making AI coding harness improvements observable and measurable.

boxdawn★ 5

Open your AI agents' black box. The waste is between your agents, not inside them. Four deterministic detectors over one trace file, priced from real usage fields. Hosted at boxdawn.com.

API-QuickCheck★ 5

⚡ The Ultimate AI Relay Forensics & Benchmark Suite. Calibrated for GPT-5.6, Claude Fable 5, Gemini 3.7 & Grok │ 4.6. Cryptographic Thinking Signature Checks, Native Stream Forensics & Anti-Ban Batch Key Testing.专治大模型中转站掺水:官方私钥签名验真 / 原生思考流鉴伪 / TTFT·TPS 流式基准 / API Key 批量检测

scaffoldscope★ 5

Controlled coding-agent harness ablations with auditable traces, reproducible evidence bundles, and SWE-bench interoperability.

ATANT★ 5

The first open evaluation framework for AI continuity. 250 narrative tests, 1835 verification questions, 10 checkpoints. Benchmark for AI memory systems, stateful agents, and long-term context persistence. No LLM in the evaluation loop.

tracelint★ 5

A deterministic linter for agent runs — reads the execution trace and flags structural bugs (ignored errors, schema violations, loops) with evidence. No LLM judge.

agentops-studio★ 5

Full-stack operations console for observable, human-in-the-loop Agent workflows.

actuate★ 5

Closed-loop graphs for LLMs: generate, judge, retry until a score, then emit a reward for training. Python package (actuate-ai) plus a control-console UI.

boteval★ 5

Chat bot evaluation

maida-ts★ 5

TypeScript mirror of the Maida trace contract. Emit and validate Maida-native agent traces from Node. Deliberately narrow -- not at feature parity with the Python package.

More in Agent frameworks & infrastructure