Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 14 of 57

usage-meter★ 16

Where every Claude token goes, live: plan-limit %, tokens and API-equivalent $ by machine, session, subagent and model. A Claude Code mod.

ringside★ 16

Free, open-source Claude Code and Codex usage tracker for Windows, macOS and Linux. Your limits, always in view. Reads local usage logs, never your login.

Usagebar★ 16

Free macOS menu bar app that shows your Claude Code, Codex, Cursor, Kimi Code, z.ai and DeepSeek usage limits: 5-hour window, weekly cap and reset times. Native, private, no account.

agentest★ 16

Agentic evals for AI agents: 165 tests, open source, offline, deterministic, no LLM judge. Reads OpenTelemetry GenAI traces, A2A and MCP. CI-ready. pip install agentest-ai

usage-meter★ 16

Where every Claude token goes, live: plan-limit %, tokens and API-equivalent $ by machine, session, subagent and model. A Claude Code mod.

RagScore★ 15

⚡️ The "1-Minute RAG Audit" — Generate QA datasets & evaluate RAG systems in Colab, Jupyter, or CLI. Privacy-first, async, visual reports.

RAGnarok-AI★ 15

Local-first RAG evaluation framework for LLM applications. 100% local, no API keys required.

costclaw-telemetry★ 15

Reduce your OpenClaw agent costs. Free real-time LLM cost tracking + dashboard. Installs in 60 seconds.

ccdash★ 15

Lightweight TUI dashboard for Claude Code token usage, agent sessions, and system resources. Go.

SWE-Chatbot-Arena★ 15

Compare chatbots pairwise via multi‑round evaluations for SE tasks.

gdpval-realworks★ 15

Benchmark LLMs on real professional tasks, not academic puzzles. YAML-driven experiment pipeline + live React dashboard for GDPVal Gold Subset (220 tasks across 11 industries).

GaRAGe★ 15

[ACL 2025] GaRAGe: A Benchmark with Grounding Annotations for RAG Evaluation.

langchain-go-vs-python★ 15

Benchmarking RAG and agentic systems in Go vs Python

agentv★ 15

Light-weight AI agent evaluation and optimization framework

ai-evals-bootcamp★ 15

Learn to evaluate AI products for production — 21 hands-on lessons on evals, metrics, fairness, agents, red teaming, and release decisions for working PMs.

guppi★ 15

all tmux sessions, all agents, one interface

ModeX★ 15

Official implementation of ACL 2026 paper, "ModeX: Evaluator-Free Best-of-N Selection for Open-Ended Generation"

agent-replay★ 15

agent-replay is a 100% local, SQLite-powered CLI tool for time-travel debugging AI agents that lets you replay execution traces, diff behavioral changes, fork runs to test fixes, and run AI-powered evaluations or safety guardrails to eliminate hallucinations and production failures.

blockhouse★ 15

A launch-control room for your local AI agents — block minifig crew, NASA-style GO/NO-GO board, real-time events from Claude Code hooks and agent logs. No cloud, no simulation.

kimi-code-usage★ 15

Kimi (Moonshot AI) Coding Plan API usage quota tracker — CLI + MCP Server + VSCode Extension

Alt_LLM_LeaderBoard★ 15

docker region:us

llm-human-feedback-collector-chat-interface-dpo★ 15

gradio region:us

Ternary-Bonsai-27B-tool-eval-bench-results★ 15

tool-eval-bench results for Prism ML Ternary-Bonsai-27B (Q2_0) — 8 trials, score 85, deployability 80

tokimeter★ 15

Local-first usage & cost meter for AI coding agents: Claude Code, Codex, Cursor, Grok Build, Hermes, opencode, Cline, and Copilot CLI. No account or telemetry.

claude-usage-mac★ 15

Claude Code usage in the macOS menu bar and as a desktop widget

core★ 15

Agentic AI monitoring and observability — OTEL traces and local log files from Copilot, Claude Code, Codex, and OpenCode. Token usage, cost, summaries, and efficiency insights. VS Code extension, npx, and Docker.

iab-actonomy★ 15

Actonomy: A taxonomy for describing and analyzing agent behavior.

clawdash★ 14

Real-time control plane for OpenClaw — monitor, manage, and configure AI agents and gateway from a single dashboard.

anchor-sdk★ 14

The official SDKs for Anchor API.

sourcemapr★ 14

Debug RAG pipelines with just 2 lines of code

RAG-Framework-Evaluation★ 14

This project aims to compare different Retrieval-Augmented Generation (RAG) frameworks in terms of speed and performance.

rag-visualizer★ 14

A visualisation tool for RAG

reservoir★ 14

Store all your LLM Interactions

PRD★ 14

PRD: Peer Rank and Discussion Improve Large Language Model based Evaluations

eval-layer★ 14

A skill that adds a rubric-based eval layer to any agent project. Framework-agnostic — generates rubric, test cases, judge prompt, and harness. Returns a weighted score plus a judge-leniency signal.

rubric-eval★ 14

Independent framework to test, benchmark, and evaluate LLMs & AI agents locally.

HalluRAG★ 14

Source code of "The HalluRAG Dataset: Detecting Closed-Domain Hallucinations in RAG Applications Using an LLM's Internal States" (arXiv: https://arxiv.org/abs/2412.17056)

rewind★ 14

Fix broken AI agents without re-running them. Fork at any step, replay from failure, prove the fix works. Works alongside Langfuse/LangSmith or standalone.

ccstats★ 14

Local-first CLI (and desktop) for token and cost analytics across AI coding agents

evolving-agent-harnesses★ 14

Evolving agent harnesses: a research program on how far N orchestrated calls of a small model can rival a frontier model. We evolve the harness (structure + prompts) with reflective optimizers + a verified-acceptance gate.

api-log★ 14

Transparent LLM gateway trace recorder · sub2api / CLIProxyAPI / new-api 的 LLM 网关流量录制器

llm-race-to-the-top★ 14

gradio region:us

claude-code-reverse★ 14

A local transparent proxy for Claude Code that captures every Anthropic API call — extract the full system prompt, track token & cache usage, and analyze tool-call patterns from a CLI or live React dashboard.

AcruxCore★ 14

Open-source LLMOps platform with prompt management, an OpenAI-compatible AI gateway, OpenTelemetry tracing, a versioned tool catalog, and LLM evaluation. Use hosted or self-host with TypeScript and Python SDKs.

swarm★ 14

Local-first control plane for AI-agent development — watch every Claude Code session live, ledger tasks/worktrees/resources, track token spend and cost, enforce rules.

muteval★ 14

Mutation testing for your LLM evals — degrade the system under test and check whether your evals would actually catch a regression.

agent-vcr★ 14

Snapshot tests for LLM tool calls — record once, catch regressions forever. Zero API cost in CI.

ompchamber★ 14

Browser console and diagnostic chamber for Oh-My-Pi (omp) agent sessions — live timeline, steering, real PTY, git/diff/wiki panels, and a panel-plugin platform. Elysia + Preact on Bun.

token-tracker★ 14

Token usage dashboard for Claude Code — tracks costs, sessions, models, and tools with real-time updates

claude-pacer★ 14

macOS menu-bar pace for your AI coding subscriptions — Claude Code, Codex, Antigravity, Cursor: will they last until their reset, and what to change if not

reservoir★ 14

Store all your LLM Interactions

critino★ 13

The LLM Agent Critique Platform.

decibench★ 13

The open testing standard for voice AI agents. Deterministic + semantic + RAG augmented evaluation. Local first. Zero telemetry.

agentgraph★ 13

Git-like branching, checkpointing, and comparison for AI agent execution paths. pip install agentgit

langscout★ 13

Open source LLM trace tool for Langchain built on MongoDB

aiusage★ 13

A very simple CLI util to help you avoid going over your limits in tools like Cursor, etc.

SWE-PR★ 13

Track Al coding assistants by GitHub pull requests

LLMCheckup★ 13

Code for the NAACL 2024 HCI+NLP Workshop paper "LLMCheckup: Conversational Examination of Large Language Models via Interpretability Tools and Self-explanation" (Wang et al. 2024)

llm-workbench★ 13

UI for testing prompts across various datasets locally

fore★ 13

The fore client package

rewardhackwatch★ 13

Runtime detector for reward hacking and misalignment in LLM agents (89.7% F1 on 5,391 trajectories).

cost-guardian★ 13

Real-time budget enforcement, cost tracking, and optimization for Claude Code

llm-energy-lab★ 13

Web application for benchmarking and comparing LLM behaviour, energy and emissions on cloud and bare metal under different inference configs, middleware scaffolds and post-training implementations

openclaw-monitor★ 13

Free open-source monitoring dashboard for OpenClaw AI agents — token usage, session tracking, 7-day trends, multi-model support

tokemon★ 13

Free, open-source Claude usage monitor for macOS & Raycast. Track token limits, burn rate, per-project costs, team budgets, and get alerts before you hit rate limits.

lumen★ 13

Real-time LLM token and cost monitor with TLS-intercepting proxy or HTTP relay; cross-platform with macOS status bar app and browser dashboard

cc-logger★ 13

Observability for Claude Code: HTTP hooks record sessions, sub-agents and tool calls to Postgres, so agent work can be inspected, compared across runs, and turned into guardrails.

Spanlens★ 13

Open source LLM observability and monitoring. Drop-in proxy for OpenAI, Anthropic, and Gemini with request logging, cost tracking, and agent tracing. Self-host with one Docker command. MIT.

paperclip-plugin-agent-usage★ 13

Track AI provider usage quotas in Paperclip — dashboard widget, usage page, agent tools

claude-workflow-viz★ 13

Visualize Claude Code dynamic workflows from your terminal — a fan-out→barrier→reduce DAG, a gantt timeline, live concurrency, and a self-contained interactive HTML report. Zero runtime deps.

agentinsight-sdk-python★ 13

AgentInsight Python SDK provides a Python client for the AgentInsight platform, supporting LLM application observability, tracing, evaluation, and prompt management.

llm-exercise-report★ 13

gradio region:us

YKS2025_LLM_Leaderboard★ 13

docker region:us

chat-leaderboard★ 13

static region:us

coding-agent-leaderboard★ 13

gradio leaderboard region:us

whoburnedmore★ 13

The whoburnedmore CLI — count what your AI coding agents really burned and submit to the public token leaderboard. Source of the npm package `whoburnedmore`.

deepseek-harness-wallet★ 13

Balance monitoring, per-session spend & token tracking, low-balance alerts, and an official recharge shortcut for DeepSeek Harness. / 余额监控和充值插件

WMB-100K★ 13

WMB-100K — The first 100,000-turn benchmark for AI memory systems

agent-pet-hub★ 13

A desktop pet that visualizes AI Agent execution states in real-time

harness-arena★ 13

Blind, LMSYS-style arena for comparing AI coding agent harnesses (Claude Code, Codex CLI, OpenClaw, Hermes, OnDemand) instead of models. Same task, same model, isolated runs , humans judge blind, results roll up into an Elo leaderboard. FastAPI + MongoDB backend.

jy-crpg-bench★ 13

A long-horizon CRPG benchmark for frontier agents. Raw 320x200 frames, Traditional Chinese, isometric navigation, twelve books to find.

MLSys26_AgenticCache★ 13

This repository contains the artifacts for the MLSys 2026 paper AgenticCache.

insight-module-for-agentic-systems★ 13

Full Fledged Observability plugin for OpenClaw

LLMReasoning★ 13

gradio trackio region:us

pacabench★ 13

local-first Benchmark Harness for LLM agents — resumable · process-isolated · proxy metrics

conduit★ 13

⚡️ local proxy for LLM token usage across providers, written in Rust

mcp-whichtool★ 13

Does the model pick the right tool? Tool-routing tests for your MCP server: run them on Claude and OpenAI models, see where each request lands, and fix the descriptions that lose.

tokenbill★ 13

Itemized cost breakdown for your Claude Code sessions. Reads local logs, no network, no config

alphatrion★ 12

⚒️ The open-source framework for LLM experiments and agent orchestration.

trustifai★ 12

TrustifAI: A Comprehensive Framework for AI Trustworthiness

lg-travel-agent★ 12

A travel agent python app built with LangGraph using GPT model from OpenAI to be run locally or on CI/CD with Github Actions

ravig-bench★ 12

Official implementation of "RAViG-Bench: A Benchmark for Retrieval-Augmented Visually-rich Generation with Multi-modal Automated Evaluation"

SWE-Agent-Arena★ 12

Compare agents pairwise via multi‑round evaluations for SE tasks.

logzip★ 12

Compress logs for LLM analysis — Rust-powered, Python API. 40-60% token savings.

claude-langfuse-monitor★ 12

Automatic Langfuse tracking for Claude Code

claude-code-usage★ 12

📊 CLI tool to track Claude Code usage, costs & token consumption. Analyze your AI coding assistant spending locally.

pincer★ 12

A fast multi-platform agent monitoring utility

PromptKit★ 12

Test, red-team, and deploy LLM applications with confidence. Multi-provider support (OpenAI, Anthropic, Gemini), MCP integration, self-play testing, and production SDK.

tokenjam-bench★ 12

Benchmarking & Evals for TokenJam

hermes-pixel-ui★ 12

Pixel-art real-time dashboard for Hermes Agent sessions, visualizing agents, tools, thinking states, and task progress inside an interactive office.

More in Agent frameworks & infrastructure