Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 14 of 57
Where every Claude token goes, live: plan-limit %, tokens and API-equivalent $ by machine, session, subagent and model. A Claude Code mod.
Free, open-source Claude Code and Codex usage tracker for Windows, macOS and Linux. Your limits, always in view. Reads local usage logs, never your login.
Free macOS menu bar app that shows your Claude Code, Codex, Cursor, Kimi Code, z.ai and DeepSeek usage limits: 5-hour window, weekly cap and reset times. Native, private, no account.
Agentic evals for AI agents: 165 tests, open source, offline, deterministic, no LLM judge. Reads OpenTelemetry GenAI traces, A2A and MCP. CI-ready. pip install agentest-ai
Where every Claude token goes, live: plan-limit %, tokens and API-equivalent $ by machine, session, subagent and model. A Claude Code mod.
⚡️ The "1-Minute RAG Audit" — Generate QA datasets & evaluate RAG systems in Colab, Jupyter, or CLI. Privacy-first, async, visual reports.
Local-first RAG evaluation framework for LLM applications. 100% local, no API keys required.
Reduce your OpenClaw agent costs. Free real-time LLM cost tracking + dashboard. Installs in 60 seconds.
Lightweight TUI dashboard for Claude Code token usage, agent sessions, and system resources. Go.
Compare chatbots pairwise via multi‑round evaluations for SE tasks.
Benchmark LLMs on real professional tasks, not academic puzzles. YAML-driven experiment pipeline + live React dashboard for GDPVal Gold Subset (220 tasks across 11 industries).
[ACL 2025] GaRAGe: A Benchmark with Grounding Annotations for RAG Evaluation.
Benchmarking RAG and agentic systems in Go vs Python
Light-weight AI agent evaluation and optimization framework
Learn to evaluate AI products for production — 21 hands-on lessons on evals, metrics, fairness, agents, red teaming, and release decisions for working PMs.
all tmux sessions, all agents, one interface
Official implementation of ACL 2026 paper, "ModeX: Evaluator-Free Best-of-N Selection for Open-Ended Generation"
agent-replay is a 100% local, SQLite-powered CLI tool for time-travel debugging AI agents that lets you replay execution traces, diff behavioral changes, fork runs to test fixes, and run AI-powered evaluations or safety guardrails to eliminate hallucinations and production failures.
A launch-control room for your local AI agents — block minifig crew, NASA-style GO/NO-GO board, real-time events from Claude Code hooks and agent logs. No cloud, no simulation.
Kimi (Moonshot AI) Coding Plan API usage quota tracker — CLI + MCP Server + VSCode Extension
docker region:us
gradio region:us
tool-eval-bench results for Prism ML Ternary-Bonsai-27B (Q2_0) — 8 trials, score 85, deployability 80
Local-first usage & cost meter for AI coding agents: Claude Code, Codex, Cursor, Grok Build, Hermes, opencode, Cline, and Copilot CLI. No account or telemetry.
Claude Code usage in the macOS menu bar and as a desktop widget
Agentic AI monitoring and observability — OTEL traces and local log files from Copilot, Claude Code, Codex, and OpenCode. Token usage, cost, summaries, and efficiency insights. VS Code extension, npx, and Docker.
Actonomy: A taxonomy for describing and analyzing agent behavior.
Real-time control plane for OpenClaw — monitor, manage, and configure AI agents and gateway from a single dashboard.
The official SDKs for Anchor API.
Debug RAG pipelines with just 2 lines of code
This project aims to compare different Retrieval-Augmented Generation (RAG) frameworks in terms of speed and performance.
A visualisation tool for RAG
Store all your LLM Interactions
PRD: Peer Rank and Discussion Improve Large Language Model based Evaluations
A skill that adds a rubric-based eval layer to any agent project. Framework-agnostic — generates rubric, test cases, judge prompt, and harness. Returns a weighted score plus a judge-leniency signal.
Independent framework to test, benchmark, and evaluate LLMs & AI agents locally.
Source code of "The HalluRAG Dataset: Detecting Closed-Domain Hallucinations in RAG Applications Using an LLM's Internal States" (arXiv: https://arxiv.org/abs/2412.17056)
Fix broken AI agents without re-running them. Fork at any step, replay from failure, prove the fix works. Works alongside Langfuse/LangSmith or standalone.
Local-first CLI (and desktop) for token and cost analytics across AI coding agents
Evolving agent harnesses: a research program on how far N orchestrated calls of a small model can rival a frontier model. We evolve the harness (structure + prompts) with reflective optimizers + a verified-acceptance gate.
Transparent LLM gateway trace recorder · sub2api / CLIProxyAPI / new-api 的 LLM 网关流量录制器
gradio region:us
A local transparent proxy for Claude Code that captures every Anthropic API call — extract the full system prompt, track token & cache usage, and analyze tool-call patterns from a CLI or live React dashboard.
Open-source LLMOps platform with prompt management, an OpenAI-compatible AI gateway, OpenTelemetry tracing, a versioned tool catalog, and LLM evaluation. Use hosted or self-host with TypeScript and Python SDKs.
Local-first control plane for AI-agent development — watch every Claude Code session live, ledger tasks/worktrees/resources, track token spend and cost, enforce rules.
Mutation testing for your LLM evals — degrade the system under test and check whether your evals would actually catch a regression.
Snapshot tests for LLM tool calls — record once, catch regressions forever. Zero API cost in CI.
Browser console and diagnostic chamber for Oh-My-Pi (omp) agent sessions — live timeline, steering, real PTY, git/diff/wiki panels, and a panel-plugin platform. Elysia + Preact on Bun.
Token usage dashboard for Claude Code — tracks costs, sessions, models, and tools with real-time updates
macOS menu-bar pace for your AI coding subscriptions — Claude Code, Codex, Antigravity, Cursor: will they last until their reset, and what to change if not
Store all your LLM Interactions
The LLM Agent Critique Platform.
The open testing standard for voice AI agents. Deterministic + semantic + RAG augmented evaluation. Local first. Zero telemetry.
Git-like branching, checkpointing, and comparison for AI agent execution paths. pip install agentgit
Open source LLM trace tool for Langchain built on MongoDB
A very simple CLI util to help you avoid going over your limits in tools like Cursor, etc.
Track Al coding assistants by GitHub pull requests
Code for the NAACL 2024 HCI+NLP Workshop paper "LLMCheckup: Conversational Examination of Large Language Models via Interpretability Tools and Self-explanation" (Wang et al. 2024)
UI for testing prompts across various datasets locally
The fore client package
Runtime detector for reward hacking and misalignment in LLM agents (89.7% F1 on 5,391 trajectories).
Real-time budget enforcement, cost tracking, and optimization for Claude Code
Web application for benchmarking and comparing LLM behaviour, energy and emissions on cloud and bare metal under different inference configs, middleware scaffolds and post-training implementations
Free open-source monitoring dashboard for OpenClaw AI agents — token usage, session tracking, 7-day trends, multi-model support
Free, open-source Claude usage monitor for macOS & Raycast. Track token limits, burn rate, per-project costs, team budgets, and get alerts before you hit rate limits.
Real-time LLM token and cost monitor with TLS-intercepting proxy or HTTP relay; cross-platform with macOS status bar app and browser dashboard
Observability for Claude Code: HTTP hooks record sessions, sub-agents and tool calls to Postgres, so agent work can be inspected, compared across runs, and turned into guardrails.
Open source LLM observability and monitoring. Drop-in proxy for OpenAI, Anthropic, and Gemini with request logging, cost tracking, and agent tracing. Self-host with one Docker command. MIT.
Track AI provider usage quotas in Paperclip — dashboard widget, usage page, agent tools
Visualize Claude Code dynamic workflows from your terminal — a fan-out→barrier→reduce DAG, a gantt timeline, live concurrency, and a self-contained interactive HTML report. Zero runtime deps.
AgentInsight Python SDK provides a Python client for the AgentInsight platform, supporting LLM application observability, tracing, evaluation, and prompt management.
gradio region:us
docker region:us
static region:us
gradio leaderboard region:us
The whoburnedmore CLI — count what your AI coding agents really burned and submit to the public token leaderboard. Source of the npm package `whoburnedmore`.
Balance monitoring, per-session spend & token tracking, low-balance alerts, and an official recharge shortcut for DeepSeek Harness. / 余额监控和充值插件
WMB-100K — The first 100,000-turn benchmark for AI memory systems
A desktop pet that visualizes AI Agent execution states in real-time
Blind, LMSYS-style arena for comparing AI coding agent harnesses (Claude Code, Codex CLI, OpenClaw, Hermes, OnDemand) instead of models. Same task, same model, isolated runs , humans judge blind, results roll up into an Elo leaderboard. FastAPI + MongoDB backend.
A long-horizon CRPG benchmark for frontier agents. Raw 320x200 frames, Traditional Chinese, isometric navigation, twelve books to find.
This repository contains the artifacts for the MLSys 2026 paper AgenticCache.
Full Fledged Observability plugin for OpenClaw
gradio trackio region:us
local-first Benchmark Harness for LLM agents — resumable · process-isolated · proxy metrics
⚡️ local proxy for LLM token usage across providers, written in Rust
Does the model pick the right tool? Tool-routing tests for your MCP server: run them on Claude and OpenAI models, see where each request lands, and fix the descriptions that lose.
Itemized cost breakdown for your Claude Code sessions. Reads local logs, no network, no config
⚒️ The open-source framework for LLM experiments and agent orchestration.
TrustifAI: A Comprehensive Framework for AI Trustworthiness
A travel agent python app built with LangGraph using GPT model from OpenAI to be run locally or on CI/CD with Github Actions
Official implementation of "RAViG-Bench: A Benchmark for Retrieval-Augmented Visually-rich Generation with Multi-modal Automated Evaluation"
Compare agents pairwise via multi‑round evaluations for SE tasks.
Compress logs for LLM analysis — Rust-powered, Python API. 40-60% token savings.
Automatic Langfuse tracking for Claude Code
📊 CLI tool to track Claude Code usage, costs & token consumption. Analyze your AI coding assistant spending locally.
A fast multi-platform agent monitoring utility
Test, red-team, and deploy LLM applications with confidence. Multi-provider support (OpenAI, Anthropic, Gemini), MCP integration, self-play testing, and production SDK.
Benchmarking & Evals for TokenJam
Pixel-art real-time dashboard for Hermes Agent sessions, visualizing agents, tools, thinking states, and task progress inside an interactive office.