Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 21 of 57
Watch every AI coding agent running on your Mac — Claude Code, Codex, Cursor, Grok Build, Grok Bot, AntiGravity, Claude Cowork, ChatGPT Work — in one live board: what you asked, who is working, who is done, who needs you.
Self-hosted AI usage and cost panel for Raspberry Pi. Tracks spend and quotas across Anthropic, OpenAI, OpenRouter, and Claude Code on a touchscreen, with a web console and Home Assistant integration. Pure-stdlib Python, zero dependencies.
Svelte 5 SPA trace viewer for api-log · LLM 网关日志查看器,可对接兼容的 JSONL / SQLite trace 存储
-
Local evidence, diagnosis, and control loop for prompts, checkpoints, and AI agents.
Open, causal evaluations for content-addressed semantics and multi-agent coordination.
Kernel latency control plane for agentic AI: DVFS, cpuidle, IRQ, scheduler, workqueue, I/O, MM, VFS, and cgroup budgets.
Local-first macOS menu bar app for tracking AI coding agent usage, transcripts, and workflows.
Testing and evaluation framework for AI agents
The AI Operations Control Plane — monitor, govern, and automate your AI agents
Monitor Codex usage and switch accounts across Codex CLI and desktop apps on macOS and Windows.
Live token usage & cost HUD inside Codex App. Renderer-injected, fully local, zero telemetry.
Live Codex quota tray panel with animated weather scenes for Windows
Orchestrator/sub-agent RCA system with a production-grade eval and observability harness — LangGraph agents, Ragas/Promptfoo evals, OTel tracing.
Tracify is an AI observability platform that helps developers monitor, debug, and optimize AI agents in production. Trace every decision, tool call, failure, retry, latency spike, and cost in one clear view.
Multi-project mission control for Hermes Agent users.
Quality and trigger-evaluation gate for reusable AI agent skills.
Track your AI subscription usage. Works with Claude Code, Codex, OpenCode and Grok.
Meridian Claims: a real multi-agent health-insurance claims-adjudication system (CrewAI + MCP + Langfuse) demonstrating production-grade LLM observability
Claude usage on a $15 desk clock: session/week/Fable %, reset times, and email/Discord/Slack alerts when your quota resets. Open firmware for the GeekMagic SmallTV-Ultra.
Mission Control for cost-conscious OpenClaw enthusiasts. Lightweight CLI/TUI tools for agent cost monitoring, task management, and configuration.
Capture, replay, diff & audit AI agent and model runs as portable, signed evidence capsules. Open-source and self-hosted — runs in your own infrastructure, from a laptop to a cluster. No SaaS, no telemetry.
Evidence-backed verification for action-taking AI agents. Compile the task into a contract once, then verify every run against real state and trajectory evidence with zero model calls.
Prices your Claude Code and Codex sessions from the transcripts on disk, shows what each agent cost over any period, and tells you in plain English what would have been cheaper. CLI, Browser UI and MCP server.
Local-first desktop analytics for AI coding agents: Claude Code, OpenAI Codex, OpenCode, Command Code and Orion App. Tracks tokens, API-equivalent costs, prompt cache, context-window fill and plan limits, and shows your live session on Discord Rich Presence. Windows, macOS and Linux; Rust, Tauri 2 and Svelte 5.
Self-hosted durable execution engine with built-in observability for AI agent runs
A desktop app for local usage, cost, throughput, and session analytics across Claude Code, Codex, OpenCode, and OpenClaw. | 在本地统一统计和分析 Claude Code、Codex、OpenCode 与 OpenClaw 的用量、费用、吞吐和会话。
Lightweight ops dashboard for OpenClaw agent teams. See who's online, what they're working on, and what happened.
DeepSeek Harness(DSH)只读执行轨迹复盘插件,支持规则分析、独立模型解读、证据定位、任务概览与运行对比。
Local pixel-art interface for real Hermes Agent sessions and tool activity.
See where your Pi tokens and cost went: local session analytics for models, tools, subagents, skills, integrations, and history.
Reproduce & measure degenerate tool-call loops in LLMs — replay a recorded conversation against any OpenAI-compatible model and see if it loops.
Performance engineering and observability for AI agents.
Runs BOTH controls (nop + oracle) across EVERY task in a coding-agent benchmark, classifies the outcome, captures evidence, and lints leaks statically. One binary, zero LLM calls.
Estimate GenAI prompt costs from a unified, auto-updated pricing table. This repo provides a tiny helper around OpenAI responses plus a parser for a curated model pricing table.
Live quota monitor, CLI/TUI daemon, and OpenTelemetry exporter for Codex/OpenAI and Claude Code subscriptions
Windows tray monitor for Claude Code and Codex usage, limits, and sessions
Cross-platform, always-on-top bar that shows what your local AI coding agents (Claude Code, Codex, OpenCode) cost you, per agent and per project. Reads only local files; no services.
This repository is for containing source codes of LogSherlock multi-agent system.
BOSSFIGHT: can a frontier LLM run a business? An end-to-end benchmark of AI business managers.
SVDO Meter is an open-source telemetry and evaluation layer for AI coding agents. Measure execution, token usage, cost, sessions, tooling behavior, and repository alignment across Codex, Claude Code, and other coding agents.
What's your vibe tax this month? Local-only receipt for what Claude Code and Codex would cost at API prices. One command, nothing uploaded.
AI usage meters on a $5 AliExpress desk clock - Claude, Ollama Cloud, Antigravity. MIT fork of click6067-ship-it/token-tv; PRs welcome for more providers & faces.
XiangqiBench: closed-loop evaluation of LLM agents on xiangqi endgames
Tracify is an AI observability platform that helps developers monitor, debug, and optimize AI agents in production. Trace every decision, tool call, failure, retry, latency spike, and cost in one clear view.
Names the harness piece behind each problem in your sessions, what it cost, and whether the fix worked.
Visual debugging, tracing, and replay for agent workflows.
LongHarness evaluates how language-model harnesses access and reason over long contexts. Its four task suites contain 200 instances that require selective retrieval, verification, and multi-step reasoning.
See what your AI coding agents really cost — tokens, spend, cache efficiency and rate limits. Works with Claude Code and Codex, more coming. Runs entirely on your machine: no account, no telemetry, no network.
Async Rust library for LLM-as-judge checks: detect hallucinated or unsupported claims in text and critique Rust code, with typed results (Pass/Fail with issues, risks, missing tests). OpenAI, Tokio.
a local-first, multi-provider tool that captures LLM API spend and exposes it to coding agents via the Model Context Protocol
Diagnose why your LLM stream returns nothing: record and analyze OpenAI-compatible SSE streams, offline, with zero dependencies.
Windows tray dashboard for CLIProxyAPI: OAuth quota windows, usage cost history, provider status. Dark JetBrains Mono UI.
Native Codex-only usage monitor for the Windows system tray
Never hit your Claude, Codex or Grok limit by surprise. Live session and weekly usage in the macOS menu bar
Codex CLI & Desktop quota at a glance. Floating widget for Windows/macOS with remaining quota, weekly limits & reset times. 轻量配额悬浮球。
A test harness for TypeSafe's Jev :Confidence audit, Hallucination judge, Jev for RAG
Open benchmark comparing web-search engines (Tavily, Exa, Firecrawl) vs LLM-native web search on quality (LLM-as-judge), cost, and latency. Reproducible, fully logged, one-file adapters.
Open, independent benchmark of web search APIs for coding agents: Exa, Parallel, Perplexity, Firecrawl, Tavily, Linkup, Brave, You, TinyFish. Scored on grounded task completion against held-out enterprise docs tickets. Search-only and search+fetch boards, model held constant.
Understand which functions and agents drive LLM spend. Self-hosted monitoring and an open model pricing catalog.
Self-hosted analytics for Claude Code subscriptions: see where your tokens go, what they'd cost on the API, and what to automate to spend less.
Watch Claude Code and Codex agents work in real time — a read-only observer with a clean event API and swappable frontends.
Verify the LLM endpoint you're actually using: model authenticity, billing inflation, relay provenance, performance and silent downgrades. Single binary, no dependencies, HTML report with CI-friendly exit codes.
Are you using Claude Code well? Finds the habits and settings that waste tokens, from the transcripts on your machine. Local dashboard, stdlib-only Python.
Fast emulator of LLM provider HTTP APIs for testing AI enabled apps without using a real LLM.
Evaluate complete agent systems: capture execution evidence, inspect reports, and compare improvements.
An evidence-based index of AI agent harnesses mapped by control function and autonomy.
Catch when your agent skills, instructions, and rules stop triggering — or stop working.
Local desktop panel that turns your AI coding sessions into an isometric dwarf mining colony - watch, message, and manage your agents in real time.
Client-side HTTPS egress proxy for TokenSaver — MITM capture for Claude Code / Cursor / agents.
Local token, context, cache, and cost telemetry for Codex conversations.
ChatGPT 降智测试 / GPT 降智检测:一句「鹈鹕骑自行车」SVG 动画提示词,1 分钟判断 ChatGPT 有没有降智(附满血样例与 12 分评分表)
Track Claude (Anthropic) usage in Home Assistant: session/weekly limits, burn rate, runway, time-to-limit and more. HACS custom integration.
Claude Haiku 5.5 as a decision model, benchmarked head-to-head against Jev and the OpenAI Decisions API
Claude Code usage stats in the desktop app: a token heat map, usage history by model, an hour-of-day chart and a Spotify Wrapped style year-in-review card.
Local test cockpit for ConnectOnion agents: create, run, QR, live logs, one-click diagnostics.
Open-source, MCP-native benchmark for whether AI agents reason reliably over fragmented enterprise knowledge — accuracy, provenance, correct refusal.
Conversation tests for AI agents: a simulated user talks to your agent through its API, a judge checks every claim, and convy reports pass rate, tokens, time and what changed between builds.
Companion code for the talk "Managing Production Agents at Scale — from Chaos to Reliability". One Google ADK agent, three production failure modes: eval-driven development, resilience, and zero-trust on Vertex AI.
Benchmark of AI coding agents building web apps and deploying them to Cloudish, DigitalOcean, Render and Vercel, measuring success, time, tokens, cost, human steps and security.
Find out why your AI agent failed, what it cost, and whether the fix worked.
macOS menu-bar indicator of Claude and Codex usage: 5-hour window, time to reset, weekly %. Liquid Glass UI, notarized, ~14 MB idle, no telemetry.
Self-hosted dashboard for Claude Code, Codex and Antigravity usage limits across all your machines and your team: 5-hour and weekly windows, spending plans, forecasts. No provider tokens needed.
See where a coding-agent session's tokens and dollars went: a local inspector for Claude Code, Codex, GitHub Copilot, Cursor, Gemini CLI, OpenCode and Hermes sessions, with a step-by-step timeline, per-step cost and analytics. Nothing leaves your machine.
Evidence, governance, and static reports for agentic runs
Live map of your local GitHub Copilot agents in the Copilot app, Copilot CLI, and VS Code: sessions, sub-agents, tools, data flows, and models. macOS menu bar app, Windows tray app, and web UI. Not an official GitHub product.
Code and dataset for 'ToolAlignBench: Investigating Alignment Conflicts in Tool-Calling Enabled LLMs' — do safety-aligned tool-calling agents obey deployment instructions? (ICML 2026 Pluralistic Alignment Workshop)
Watch your Claude Code (and Codex) agents live as pixel characters: handoffs, cost, loops, stuck permissions, context pressure — and familiars that evolve with good habits.
Count tokens across AI models and get concrete, priced suggestions to cut your prompt costs. Runs offline.
A standalone mock LLM server for deterministic testing: OpenAI, Anthropic, and Responses API formats
Claude Code and ChatGPT (Codex) subscription limits in one card: a KDE Plasma 6 widget and a Rainmeter skin for Windows.
See what your coding agents actually do and spend: sessions, context and cost for Claude Code, Codex, OpenCode, DeepSeek Harness, WorkBuddy. Local-first.
Generic DeepSeek Harness (dsh) plugins: A2A protocol server, session storage mirror, and Langfuse observability.
Purpose-aware AI agent benchmark: 72 synthetic scenarios, four reproducible baselines and inspectable permission traces. Python, offline.
Find the LLM calls your AI agent may never have needed.
Profile: proof-driven AI systems, evaluation, and agent reliability
Real-time monitoring dashboard for OpenClaw AI Agent — track tool calls, conversations, token usage & costs locally. Zero dependencies, MIT license.
Cut your OpenClaw / ZeroClaw token bill. Find which model earns its cost. Prove whether optimizations actually work. Local, no upload.
A minimal sandbox to run, score, and compare AI agent outputs locally.
CLI for evaluating AI agent harnesses across GitHub Copilot, Claude Code, Codex, and custom tools.