Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 16 of 57

agent-pulse★ 10

Know what your agents are doing

aether★ 10

Open-source, local observability TUI for Claude Code and Codex — context, cost, tokens, agents, tools, compactions, and code changes.

Meterbility★ 10

The debugger for AI agents — capture every run, inspect every decision, pause and inject live, fork from any step.

suanli-dashboard★ 10

macOS 悬浮仪表盘:监控 OpenAI Codex 与 Claude Code 的订阅额度余额、Token 消耗与多设备趋势,支持 Touch Bar 常驻显示

codex-notch★ 10

Native macOS notch activity for ChatGPT Codex tasks and weekly quota

BlackFlare★ 10

BlackFlare - A native macOS menu bar app for Claude Code and Codex. Keep your Mac awake during long runs, get pinged when a session finishes or needs you, switch your defaults from the menu, and watch plan usage — all on-device.

skill-tracker★ 10

Local-first dashboard for visualizing AI-agent skill calls, Chinese skill descriptions, and duplicate-skill governance.

reeve★ 10

A terminal cockpit for AI agents. Observe, evaluate, intervene.

aiusage★ 10

Cross-platform (Linux/macOS/Windows) terminal + tray usage tracker for Claude Code, Codex, and other AI coding subscriptions

cctrace★ 10

See exactly what your coding agent does on the wire. TLS-intercepting tracer for the Claude Code, Codex, Grok and Kimi CLIs: full first-party capture in a live web UI — reconstructed sessions, session replay, find-in-session, per-turn cost/cache/latency, and a cross-project dashboard of every run.

polylogue★ 10

A local evidence system for AI work—unifying ChatGPT, Claude, Codex, Gemini, and agent histories into one searchable, auditable archive.

ccx★ 10

Session viewer for coding agents — browse, search, trace, and export Claude Code, Codex, and Grok conversations

relayroom★ 10

Self-hosted coordination and observability hub for AI coding agents

agy_watch★ 10

Antigravity Observability Console & Wire-Tap Module

hindcast★ 10

Browse, search & replay every Claude Code session on your Mac. Local-first session viewer, timeline scrubber & cost tracker (ccusage alternative).

OpenEval★ 10

Free, local-first evaluation dashboard for agent CLIs with cross-harness transcript collection, live trace intelligence, deterministic graders, and accuracy audits.

agent-think-map★ 10

Watch any AI agent think. Embeddable visual tracing for chain-of-thought, skills, tool calls, and MCP - live inside the chat UI you already ship. No account, no API key.

ai-usage-extension★ 10

Privacy-first Chrome extension for tracking Claude, ChatGPT, and Codex usage limits with a popup, toolbar badge, and on-page overlay.

FP-AMB★ 10

First-Person Agent Memory Bench. 10 Categories including fact recall, multi-hop links, temporal reasoning, fact overwrites, speaker traps, refusal, credibility, and agentic tool usage. 540K token / 60 session corpus, all in first person. Dynamic output-answer-key portion. Comprehensive report with visuals and miss breakdown.

clawdeck★ 10

Watch Claude Code work: a zero-dependency local dashboard for sessions, events, cost, worktrees, and reviews - one loopback server, no build step.

mac-agent-monitor-app★ 10

MAC版灵动岛--Agent运行监测:本地查看 Codex、Claude Code、VS Code 等 AI Agent 的运行状态、工作时长、对话与 Token 用量。

workisland★ 10

WorkIsland — macOS 本地优先的 AI Agent 状态中枢:查看任务、处理审批并一键跳回上下文。

memory_and_context_eval_harness★ 10

Open evaluation harness for AI agent memory systems. Runs LoCoMo and LongMemEval against Synap, Mem0, Zep and Supermemory with pluggable provider adapters.

agent-hitch★ 10

The execution layer for model and harness evaluation.

mocktopus★ 10

🐙 Multi-armed mocks for LLM apps - Drop-in replacement for OpenAI/Anthropic APIs for deterministic testing

rag-eval-workflow★ 10

A framework-neutral RAG evaluation skill with golden sets, regression gates, and script-domain checks.

halu-core★ 10

Scores whether an autonomous AI agent actually did the work or just hallucinated its report, by checking every claim against recorded API calls. The core engine behind NoHalu.

gauntlet-loop-brain★ 10

Evidence-gated Obsidian brain for bounded multi-agent improvement loops.

harness-bench★ 10

Benchmark leaderboard for AI coding harnesses by model.

ctxfire★ 10

Static cost analyzer for multi-agent context graphs: explain what agents load and estimate tokens per fire and per day.

factual-lookup-company-news-search★ 10

Open, independent benchmark on web search APIs for AI agents - factual lookup on 300 company news questions, ranked by cost per 1,000 correct answers. TinyFish, Parallel, Perplexity, Linkup, Firecrawl, Brave, You, Exa, Tavily, Google SERP.

silo★ 10

Local-first simulation and evaluation framework for AI agents. Define worlds as TypeScript, run agents inside them, score what they did.

dsh-budget★ 10

Cost governance for DeepSeek Harness: token/cost metering per model, session and day, budget caps with threshold alerts and over-limit policies, carbon footprint estimation, per-model latency benchmarks, a Settings budget tab, and the /budget command. Maintenance status: FROZEN since 2026-10-05.

trail★ 10

Laravel AI SDK observability. Monitor agent runs, traces, tool calls, token usage, and estimated costs with a self-hosted dashboard.

agent-lookout★ 10

A local dashboard for the AI agent sessions already running on your machine. Starts nothing, and sends nothing unless you set up email or a webhook.

herdr-lcars★ 10

Command up to 2,000 Herdr AI agents from one LCARS bridge, track Claude/Codex quota per account, and hand off verified context without losing work.

dsh-fast★ 10

Read-only performance diagnostics for DeepSeek Harness: session load/restore timing, spill-hit counts, compaction count and trigger, context-injection volume (AGENTS.md/skills/tool-schema token share), and LLM cache hit rate — surfaced via /fast, persisted as reconstructable session events with async sampling off the model path.

ai-poker-arena★ 10

Making multiple LLMs play Texas Holdem against each other

titration★ 10

Let your coding agent fix a prompt until it actually works, and prove it did. A self-hosted MCP server: cross-vendor judges, frozen baselines and memory.

QUACK★ 10

🦆 [EMNLP 2026] Official Implementation of QUACK: Questioning, Understanding, and Auditing CollaborativeKnowledge in Multimodal Social Deduction Agents

AI-Agent-Evals-Course★ 10

Learn AI Evals with Manjeet

wyrd★ 10

Build verification workflows that fit your systems. Declare requirements, observe real behavior, and verify outcomes with open-source building blocks.

Lumina★ 10

A lightweight observability platform for LLM applications. Track costs, latency, and quality across your AI systems with minimal overhead.

token-speed-patrol★ 10

基于 GitHub Actions 的 LLM 测速看板,含 OpenRouter, Nvidia NIM, OpenCode 等免费模型实时测速数据

claude-theater★ 10

Watch your Claude Code subagents work — a live office for every conversation.

tenbri★ 10

The open-source Agent Decision Record: every decision signed, chained and provable.

roving-office★ 10

An office full of characters, and the characters are your AI agents. A soft-3D isometric room that renders live agent sessions as people who walk in, sit down, look things up and clock off.

spare★ 10

Free, open-source macOS menu bar app that shows CPU and memory use by app, project, and AI agent

aura★ 10

The runtime coat around agent loops.

jev-state★ 10

Build and regression-test conversational state machines powered by Jev. Inspect decisions, capture failing conversations as tests, and export runnable TypeScript for your app.

rag-agent-failure-catalog★ 10

Catalog of silent failures in RAG systems and AI agents, with reproduce, detect and mitigate code

guard★ 10

Observability and guardrails for your ai agents. AgentTrail Guard traces your agent's execution and automatically compiles repeat failures into permanent guardrails.

ClawProbe★ 9

Real-time observability for OpenClaw agents — token usage, API cost, context health, and smart alerts. Zero config. 100% local.

llm-meter★ 9

Terminal-based monitoring tool for LLM API usage and costs

kakveda★ 9

Open-source failure intelligence platform for LLM & agent systems. Adds failure memory, pre-flight warnings, pattern detection, and system health scoring.

quota-tracker★ 9

High-precision quota and token monitoring for AI Agents. Real-time analytics and usage tracking for Claude, Codex, Gemini, and Copilot to prevent API exhaustion and optimize LLM budgets.

Shadow★ 9

Behavior contracts for AI agents

coreax-sdk★ 9

Open-source Runtime Assurance SDK for AI agents in production

ACP-inspector★ 9

Debug and validate Agent Client Protocol (ACP) traffic — protocol conformance testing for AI coding agents.

mission-control-dashboard★ 9

Mesh-first real-time dashboard for Claude Code, Hermes, OpenClaw, MLX, and local memory/routing systems.

openclaw-pixel-agents-dashboard★ 9

🎮 Pixel art ops dashboard for OpenClaw AI agents. Watch your agents work in real-time as character sprites in a shared office — with live activity bubbles, hardware monitoring, service controls, and an embedded task spawner with mini-chat.

langchain-fixie-marvin★ 9

We compared LangChain, Fixie, and Marvin

fluxloop★ 9

Open-source toolkit for running reproducible, offline-first simulations of AI agents against dynamic scenarios

rasa-rest-api-loadtest★ 9

Basic tool (based on Locust.io) to perform load test of Rasa based chatbot via REST API.

assistant-cross-validation★ 9

Python script to cross-validate an assistant workspace

lunary-js★ 9

Lunary AI JS Client (analytics and logging for AI apps) 📈

api-token-usage-alert★ 9

Stop surprise API bills before they happen. API Spend Guard is a read-only, multi-provider cost monitoring platform for teams using OpenAI, Anthropic, GCP, and other AI/API services. It aggregates spend into one dashboard and sends real-time alerts when usage crosses your custom thresholds.

PromptDrifter★ 9

PromptDrifter – one‑command CI guardrail that catches prompt drift and fails the build when your LLM answers change.

llm-prompt-testing★ 9

Prompt Testing framework for LLMs. Compute NLP and Responsible AI metrics for each model-generated answer.

chatgpt-token-cost-analysis★ 9

Python script and HTML page to analyze token costs from ChatGPT export chats. Extracts messages, calculates token usage, and determines monthly costs. The Python script saves results to a CSV file, while the HTML page provides an interactive, local analysis tool with support for multiple models and ensures data privacy.

rompt★ 9

GPT prompt A/B testing built with Next.js 13 and OpenAI

Prompture★ 9

Prompture is an API-first library for requesting structured JSON output from LLMs (or any structure), validating it against a schema, and running comparative tests between models.

claude-code-plan-usage★ 9

A simple Web app optimized for TV to show Claude Code usage

RagDoctor★ 9

Make RAG reliable for enterprise use ✨

Satya_AI★ 9

A lightweight open-source platform by AnkTechSol for managing autonomous AI agents in real-time. Satya provides a clean dashboard and Python SDK to track AI agent tasks, monitor progress, build and curate knowledge bases from web content, and maintain full audit logs, all without complex infrastructure. Designed to be the single source of truth

agentprobe★ 9

pytest for AI Agents — Record, test, and secure AI agents. 35+ assertions, prompt injection fuzzing, cost tracking. Local-first.

K9Audit★ 9

Engineering-grade causal audit infrastructure for AI agents

chatbot-coffee-vercel★ 9

Code for an AI assistant that answers questions about coffee to demo instrumentation of GenAI apps with open-source Monocle.

biological-alignment-gridagents-benchmarks★ 9

Safety challenges for RL and LLM agents' ability to learn and properly apply biologically and economically aligned utility functions. The benchmarks are implemented in a gridworld-based environment. The environments are relatively simple, just as much complexity is added as is necessary to illustrate the relevant safety and performance aspects.

LLM-as-a-Broken-Telephone★ 9

[ACL 2025] Code implementation for the "LLM as a Broken Telephone: Iterative Generation Distorts Information" paper.

teamcity-ai-agent-testing-demo★ 9

End-to-end TeamCity framework to run AI agents on SWE-Bench Lite. Spin up isolated Docker images per task, extract patches, score with the official harness, and aggregate success rates. As an example, we'll look at Junie and Google Gemini CLI

evaluate★ 9

Evaluate - The Robust LLM Testing Framework 🦀

azureptucalc★ 9

Free, open-source Azure OpenAI PTU calculator. Compare PAYGO, PTU, Spillover, and Priority Processing pricing with live Azure API data. Prevent 429s with built-in optimization tools.

token-counter★ 9

Token counter for Anthropic

riva★ 9

Local-first observability and control plane for AI agents.

awesome-agent-benchmarks★ 9

A curated collection of the world’s most advanced benchmark datasets for evaluating Large Language Model (LLM) Agents.

claudewatch★ 9

AgentOps for Claude Code. Monitors AI agent behavior during development: real-time error loops, drift detection, friction analysis. Claude gets self-awareness hooks + MCP tools. You get cost-per-commit, agent success rates, exportable metrics. Like DevOps, but for AI agents.

AgentFlow★ 9

Monitor any AI agent system. Auto-detects failures, sends alerts. Zero config, zero runtime dependencies in core.

agent_debugger★ 9

Local-first agent debugger with replay, failure memory, smart highlights, and drift detection.

claude-usage-dashboard★ 9

Find out what your Claude Code subscription is actually worth. Self-hosted dashboard for tracking token usage, API costs, quota utilization, and session analytics.

SCARF★ 9

SCARF (System for Comprehensive Assessment of RAG Frameworks) is a modular evaluation framework for benchmarking deployed Retrieval Augmented Generation (RAG) applications. It offers end-to-end, black-box assessment across multiple configurations, supports automated testing with several vector databases and LLMs.

gastown-dashboard★ 9
agenttel-sdk★ 9

Agent-ready telemetry SDK — enriches OpenTelemetry across Java, Go, Python, Node.js, and browser with structured context for AI-driven observability.

codeledger★ 9

Token-level cost intelligence for AI coding tools — budget alerts, anomaly detection, and multi-tool tracking across Claude Code, Codex CLI, Cline, and Gemini CLI

tlm★ 9

Score the trustworthiness of outputs from any LLM in real-time

benchclaw★ 9

BenchClaw — Multi-dimensional AI agent evaluation with 17-judge AI Tribunal, 10 scoring dimensions, radar charts, and deception detection. Benchmark any LLM agent.

agentlens★ 9

🔍 Chrome DevTools for AI Agents. Time-travel debugging, cost tracking, and anomaly detection for multi-agent LLM workflows. MCP-native.

Local-LLM-Arena★ 9

Privacy-first local AI model comparison platform with blind evaluation, per-model hyperparameters, and multi-configuration testing. Compare 2-6 models side-by-side through Ollama with zero cloud dependencies.

umwelten★ 9

CLI tool for evaluating and comparing AI models across Google, Ollama, OpenRouter, LM Studio, LlamaBarn, and GitHub Models. Features robust error handling, cost tracking, memory-augmented chat, and dynamic test coverage.

brainusage★ 9

GNOME Shell extension that tracks Claude and Codex AI usage limits in the top panel

More in Agent frameworks & infrastructure