Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 10 of 57

ARGUS★ 37

Catch silent failures in AI agents before your users do

Anamnesis★ 37

A local CLI for logging predictions before the outcome and checking whether your "80% sure" comes true 80% of the time.

sse-agent-bench★ 37

Configuration-driven evaluation, regression testing, replay, and load testing for SSE-streaming LLM agents

frugent★ 37

Cost-efficient coding-agent runtime: prefix-aware prompt-cache routing, poisoning-safe semantic cache, budgeted context compression, request batching/dedup, cascading model routing with DP-optimal policies, hard budget scheduling, span-level cost attribution, sandboxed tools, synthetic workload factory. NumPy core, CPU-only torch extra.

AgentIF-OneDay★ 36
scout_apm_elixir★ 36

ScoutAPM Elixir Agent. Supports Phoenix and other frameworks.

CostBench★ 36

The official code repository for the paper "CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agents"

Qworld★ 36

Qworld: Question-Specific Evaluation Criteria for LLMs

japanese-chatbot-arena-leaderboard★ 36

streamlit region:us

Multi-User-LLM-Agent★ 36

Official code for the paper: "Multi-User Large Language Model Agents"

TwinRouterBench★ 36

Per-step LLM routing benchmark with 970 static labels, live SWE-bench evaluation, an open data pipeline, and a public leaderboard.

agent-interp-envs★ 36

A framework for testing and evaluating AI agents across various task domains, designed for misalignment interpretability research.

jaz-evals★ 36

Reproduction package for "Harness as a Language: A Minimalist Agent Framework With Maximal Expressivity" — ten agent arms across AppWorld and StuLife

llm-usage-metrics★ 36

Token and cost reports for 17 AI coding agents (Claude Code, Codex, Gemini CLI, OpenCode, Copilot, pi and more), read from local session logs across all your machines

TRACES.md★ 36

TRACES.md — AGENTS.md, but for what agents did. A simple, open format for recording what AI agents did, for error analysis and self-optimization.

JudgeIt-LLM-as-a-Judge★ 35

Automation Framework using LLM-as-a-judge to evaluate of Agentic AI, RAG, Text2SQL at scale; that is a good proxy for human judgement.

Traceplane★ 35

Multi-Agent Continuity Layer — Keep your AI agents on a single, observable, handoff-ready work thread

gridwatch★ 35

A Tron-themed desktop app that visualises your GitHub Copilot CLI sessions — browse history, track token usage, and explore activity across all your coding sessions.

ruby_llm-contract★ 35

Validate and retry LLM outputs for ruby_llm. Describe the JSON response you expect, fall back to a stronger model when the cheaper one fails the rules, and gate CI on regressions — all as one contract object per step.

SkillEval★ 35

A visual workbench for A/B testing AI skills. Upload two skill files, run them through a batch of test prompts, and let an AI judge score the results.

dbt-llm-evals★ 35

The warehouse-native LLM evaluation package for dbt™ - monitor AI quality without data egress

GeoBenchX★ 35

LLM-agents benchmark set of geospatial tasks requiring multi-step tool use; and LLM-as-Judge based evaluation framework.

hermes-ai-infrastructure-monitoring-toolkit★ 35

Autonomous AI infrastructure monitoring using Hermes Agent. Cron-based research ingestion, cost forecasting, and headless systemd deployment.

ICRTL-Benchmark★ 35

ICRTL Benchmark: Industrial-level RTL design challenges for evaluating PPA optimization, code generation, and LLM applications in EDA.

claudetuner★ 35

Know when you'll hit your Claude, ChatGPT & Gemini usage limits — live 5h/weekly gauges, reset forecasts, and the usage history they don't show you. Covers Claude Code via the shared subscription pool. Chrome extension + dashboard, MIT.

gridwatch★ 35

A Tron-themed desktop app that visualises your GitHub Copilot CLI sessions — browse history, track token usage, and explore activity across all your coding sessions.

tokenpulse★ 35

TokenPulse — Local-first AI agent token usage monitor (Windows/macOS/Linux). 本地优先的多 Agent 消耗监控:实时看板·趋势·成本估算,支持 Claude Code/Codex/Gemini/Qwen/ZCode,数据不出本机。

agent-monitor★ 35

A live tree of the Codex and Claude Code sessions, subagents and teammates working in a repository

MASEval★ 34

Multi-Agent LLM Evaluation Docs: https://maseval.readthedocs.io/

rurage★ 34
n8n-nodes-ai-agent-langfuse★ 34

Custom n8n node for integrating Langfuse with AI agent workflows. Provides logging and tracing support for LLM-powered automation in n8n.

MultiAgentVerification★ 34

Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifiers

claudestat★ 34

Real-time execution trace and cost intelligence for Claude Code

github-copilot-insights-dashboard★ 34

A centralized analytics dashboard for enterprise GitHub Copilot adoption. Visualize usage metrics, license allocation, premium request consumption, and AI model activity — all sourced from the GitHub Copilot Usage Metrics API.

Promptrace★ 34

Lightweight, dependency-free logging and analytics for LLM API calls — track tokens, cost, and latency to a plain JSONL file.

mizan-llm-leaderboard★ 34

gradio region:us

ukrainian-llm-leaderboard★ 34

gradio region:us

Agentic-Tool-Optimization★ 34

Local-first multi-runtime AI cockpit (CLI + desktop + MCP). War-rooms, review receipts, open multi-model coding benchmarks. MIT.

argus★ 34

Argus is a desktop app that helps find and fix wasted agent work. Local, free and open source. Focuses on business tasks, not code. Works with Claude Cowork / Chat / Code, ChatGPT Work and Codex.

vero★ 34

VeRO is an evaluation harness for using coding agents to optimize LLM-based agents and workflows. It treats agent code as a versioned artifact — making changes, evaluating results, and hill-climbing toward better performance using git version control.

ceiling★ 34

A focused Windows AI capacity monitor for Codex, Claude, Cursor, Grok, Gemini, OpenCode, and Copilot.

ConsumptionCentral-for-Microsoft-Copilot★ 34

One Power BI template for Microsoft Copilot credit consumption and cost - Cowork/Work IQ, Copilot Studio, GitHub Copilot and Azure AI Foundry.

IdeaTaste★ 34

IdeaTaste - A benchmark for research taste: let LLM agents propose research ideas, then check how many of them the research area actually went on to do.

shinzo★ 33

Complete observability platform for AI agents and MCP servers. Improve your AI deployment outcomes, identify inference inefficiencies, and gain insights into real agent usage patterns.

freshstack★ 33

This repository helps you evaluate your models on the FreshStack benchmark!

llm-pricing★ 33
function-calling-leaderboard-for-zhtw★ 33

Function Calling Leaderboard for Traditional Chinese (zh-tw)

clemcore★ 33

A Framework for the Systematic Evaluation of Chat-Optimized Language Models as Conversational Agents and an Extensible Benchmark

TIME★ 33

[NeurIPS 2025 (Spotlight✨)] TIME: A Multi-level Benchmark for Temporal Reasoning of LLMs in Real-World Scenario

straude★ 33

Strava for Claude Code / Codex. CLI to track your AI coding output tokens, spend, and streaks. Compete on a global leaderboard and see what your friends are building. Make every agentic coding session count.

RiskLab★ 33

RiskLab (ACL 2026 Demo): A Toolkit for Probing Emergent Risks in LLM-Based Multi-Agent Systems

Lupen★ 33

Itemized cost receipts for Claude Code and Codex — by turn, verified, local.

retok★ 33

Token-efficiency analyzer for AI coding agents (Claude Code, OpenAI Codex CLI): cost estimates, cache analysis, and savings advice. Zero dependencies.

squaredev★ 32

The platform for developers building applications powered by LLMs

function-calling-eval★ 32

The DPAB-α Benchmark

llm-observability-opik★ 32

LLM Evaluation and Observability System for Football Content

Agentic-CIGEval★ 32

Code of our paper "A Unified Agentic Framework for Evaluating Conditional Image Generation".

vero-eval★ 32

Open source framework for evaluating AI Agents

LatencySensitiveBench★ 32

First Latency-Aware Competitive LLM Agent Benchmark

agent-super-spy★ 32

Full stack observability for local AI agent development. Provides a llm-proxy to use as LLM provider base URL, logs traces to opik.

recovery-bench★ 32

Recovery-Bench is a benchmark for evaluating the capability of LLM agents to recover from mistakes

episodiq★ 32

Economical human-readable logs and structural (event pattern) retrieval for agentic trajectories

llm-proxy★ 32

Transparent logging proxy for LLM API traffic that auto-configures clients and records every request and response to Claude, OpenAI, and other providers for debugging, auditing, and analysis.

tracecraft★ 32

The black box for AI agents — record every session, referee every task claim. Live mirroring + atomic coordination in a bucket YOU own. No server.

maida★ 32

Pre-merge behavioral regression gate for AI agents. Check agent changes, inspect regressions, and block broken behavior before merge. Local-first.

Part_HackBench★ 32

A stress-testing benchmark for partial-completion evaluators in stateful multi-turn agents, focusing on temporary progress, rollback, and attribution failures.

Every-Other-Token★ 32

See how sure an LLM is about each word.

dsh-plugin★ 32

OpenTelemetry tracing for DeepSeek Harness (dsh): turns each agent turn into a GenAI span tree — steps, LLM calls with TTFT, tool executions, token usage — exported over standard OTLP to Jaeger, Grafana Tempo, SigNoz, Langfuse, or any compatible backend.

BurnRate★ 32

⚡ Terminal HUD, hard budget breakers & prompt-cache leak alerts for AI coding agents (Claude Code, Cursor, Codex)

llm-leaderboard-csv★ 31

CSVs of the Huggingface and LMArena LLM leaderboards, along with the code to generate them in Python and R.

arrakis★ 31

Arrakis is a library to conduct, track and visualize mechanistic interpretability experiments.

gastown-viewer-intent★ 31

Real-time agent observability dashboard. View beads tasks, agent activity, and system health in TUI or React web interface. Go + React.

claude-code-lens★ 31

Local monitor for Claude Code API traffic, logs, prompts, and tools

AgentWatch★ 31

AgentWatch — Real-time reasoning auditor and observability platform for AI agents. Catches silent failures in your agent's reasoning chain before they execute — not after they break production.

pi-quotas★ 31

Quota monitoring for the Pi coding agent. Anthropic, Codex & Copilot and OpenRouter usage at a glance

auditable★ 31

Audit any agent decision across its past, present, and future, on one typed graph.

skillcheck★ 31

Blind A/B testing for AI agent skills, .cursorrules, CLAUDE.md, and system prompts. Detects placebo and harmful instructions.

aireceipts★ 31

Your AI coding agent just billed you. Here's the receipt. Cost receipts for Claude Code, Codex, Cursor, Gemini & opencode — no account, no upload.

AgentSkill_Survey★ 31

[EMNLP2026] Agent Skill Evaluation and Evolution: Frameworks and Benchmarks

llm-mock★ 31

Self-hosted OpenAI-compatible mock server — test your LLM integrations without exposing your API key. By Axium.

plumbline★ 31

The reliability & architecture analyzer for LLM and agentic systems. By ActaClad.

ml-mmtoolsandbox★ 31

MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents

laravel-spectra★ 31

Observability for AI and LLM calls in Laravel. Automatically tracks requests to OpenAI, Anthropic, Gemini, Mistral and more, calculates token and media costs, enforces per-user budgets, and ships a real-time dashboard with OpenTelemetry export.

agent-claimcheck★ 31

Catch AI agents that say "done" when the task isn't. Checks agent-trace/v1 traces with claim-evidence rules, LLM judges and a trained classifier, returns verified / false_success / unverifiable with calibrated confidence, and ships a 300-trace labelled benchmark.

TraceAI★ 31

Open-source observability for AI applications: every LLM call, prompt, token and agent decision.

thumb★ 30

A simple prompt testing library for LLMs.

laravel-ai-guard★ 30

Laravel AI Guard 🛡️💰🤖 - Control and optimize AI costs in Laravel AI SDK applications 🚀 Track OpenAI & LLM token usage 📊, estimate AI costs before execution ⚠️, enforce AI budgets 🧾, and prevent unexpected billing spikes in production 💥.

opensmith★ 30

The open-source, local-first alternative to LangSmith. No cloud. No setup.

reticle★ 30

Postman for AI - design, evaluate, and debug LLM interactions with full transparency.

obz-cli★ 30

Multi-backend observability CLI for metrics, logs, and traces — unified interface, AI-Agent friendly

LLM-Training-Cost-Calculator★ 30

gradio region:us

domain_llm_leaderboard★ 30

gradio region:us

LLM-DetectAIve★ 30

gradio region:us

n8meter★ 30

Open-source cost metering for n8n AI workflows — token usage per workflow, priced daily, on your own machine.

Agentic_Evals★ 30

Practical evaluation techniques, benchmarks, and examples for production AI agents: RAG, tool use, planning, memory, safety, reliability, and cost.

office-for-claude-agents★ 30

Real-time agent visualization in pixel office for Claude CLI and Claude macOS app with basic Codex support /// Real time визуализация ИИ агентов в виде пиксельного офиса для Claude CLI и Claude macOS app с базовой поддержкой Codex.

MASRubric★ 30

MASRubric: Auditing Information Flow in Multi-Agent Systems with Failure-Distilled Pitfall Rubrics.

AdaMAST★ 30
openAI-dashboard★ 29

Dashboard that allows the user to visualize in a clear and intuitive way how the OpenAI models are being used.

openai-image-token-calculator★ 29

A calculator to estimate the number of tokens and cost associated with processing images using Azure OpenAI's multimodal models, including GPT-5.x, o-series, and image generation models

More in Agent frameworks & infrastructure