Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 10 of 57
Catch silent failures in AI agents before your users do
A local CLI for logging predictions before the outcome and checking whether your "80% sure" comes true 80% of the time.
Configuration-driven evaluation, regression testing, replay, and load testing for SSE-streaming LLM agents
Cost-efficient coding-agent runtime: prefix-aware prompt-cache routing, poisoning-safe semantic cache, budgeted context compression, request batching/dedup, cascading model routing with DP-optimal policies, hard budget scheduling, span-level cost attribution, sandboxed tools, synthetic workload factory. NumPy core, CPU-only torch extra.
ScoutAPM Elixir Agent. Supports Phoenix and other frameworks.
The official code repository for the paper "CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agents"
Qworld: Question-Specific Evaluation Criteria for LLMs
streamlit region:us
Official code for the paper: "Multi-User Large Language Model Agents"
Per-step LLM routing benchmark with 970 static labels, live SWE-bench evaluation, an open data pipeline, and a public leaderboard.
A framework for testing and evaluating AI agents across various task domains, designed for misalignment interpretability research.
Reproduction package for "Harness as a Language: A Minimalist Agent Framework With Maximal Expressivity" — ten agent arms across AppWorld and StuLife
Token and cost reports for 17 AI coding agents (Claude Code, Codex, Gemini CLI, OpenCode, Copilot, pi and more), read from local session logs across all your machines
TRACES.md — AGENTS.md, but for what agents did. A simple, open format for recording what AI agents did, for error analysis and self-optimization.
Automation Framework using LLM-as-a-judge to evaluate of Agentic AI, RAG, Text2SQL at scale; that is a good proxy for human judgement.
Multi-Agent Continuity Layer — Keep your AI agents on a single, observable, handoff-ready work thread
A Tron-themed desktop app that visualises your GitHub Copilot CLI sessions — browse history, track token usage, and explore activity across all your coding sessions.
Validate and retry LLM outputs for ruby_llm. Describe the JSON response you expect, fall back to a stronger model when the cheaper one fails the rules, and gate CI on regressions — all as one contract object per step.
A visual workbench for A/B testing AI skills. Upload two skill files, run them through a batch of test prompts, and let an AI judge score the results.
The warehouse-native LLM evaluation package for dbt™ - monitor AI quality without data egress
LLM-agents benchmark set of geospatial tasks requiring multi-step tool use; and LLM-as-Judge based evaluation framework.
Autonomous AI infrastructure monitoring using Hermes Agent. Cron-based research ingestion, cost forecasting, and headless systemd deployment.
ICRTL Benchmark: Industrial-level RTL design challenges for evaluating PPA optimization, code generation, and LLM applications in EDA.
Know when you'll hit your Claude, ChatGPT & Gemini usage limits — live 5h/weekly gauges, reset forecasts, and the usage history they don't show you. Covers Claude Code via the shared subscription pool. Chrome extension + dashboard, MIT.
A Tron-themed desktop app that visualises your GitHub Copilot CLI sessions — browse history, track token usage, and explore activity across all your coding sessions.
TokenPulse — Local-first AI agent token usage monitor (Windows/macOS/Linux). 本地优先的多 Agent 消耗监控:实时看板·趋势·成本估算,支持 Claude Code/Codex/Gemini/Qwen/ZCode,数据不出本机。
A live tree of the Codex and Claude Code sessions, subagents and teammates working in a repository
Multi-Agent LLM Evaluation Docs: https://maseval.readthedocs.io/
Custom n8n node for integrating Langfuse with AI agent workflows. Provides logging and tracing support for LLM-powered automation in n8n.
Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifiers
Real-time execution trace and cost intelligence for Claude Code
A centralized analytics dashboard for enterprise GitHub Copilot adoption. Visualize usage metrics, license allocation, premium request consumption, and AI model activity — all sourced from the GitHub Copilot Usage Metrics API.
Lightweight, dependency-free logging and analytics for LLM API calls — track tokens, cost, and latency to a plain JSONL file.
gradio region:us
gradio region:us
Local-first multi-runtime AI cockpit (CLI + desktop + MCP). War-rooms, review receipts, open multi-model coding benchmarks. MIT.
Argus is a desktop app that helps find and fix wasted agent work. Local, free and open source. Focuses on business tasks, not code. Works with Claude Cowork / Chat / Code, ChatGPT Work and Codex.
VeRO is an evaluation harness for using coding agents to optimize LLM-based agents and workflows. It treats agent code as a versioned artifact — making changes, evaluating results, and hill-climbing toward better performance using git version control.
A focused Windows AI capacity monitor for Codex, Claude, Cursor, Grok, Gemini, OpenCode, and Copilot.
One Power BI template for Microsoft Copilot credit consumption and cost - Cowork/Work IQ, Copilot Studio, GitHub Copilot and Azure AI Foundry.
IdeaTaste - A benchmark for research taste: let LLM agents propose research ideas, then check how many of them the research area actually went on to do.
Complete observability platform for AI agents and MCP servers. Improve your AI deployment outcomes, identify inference inefficiencies, and gain insights into real agent usage patterns.
This repository helps you evaluate your models on the FreshStack benchmark!
Function Calling Leaderboard for Traditional Chinese (zh-tw)
A Framework for the Systematic Evaluation of Chat-Optimized Language Models as Conversational Agents and an Extensible Benchmark
[NeurIPS 2025 (Spotlight✨)] TIME: A Multi-level Benchmark for Temporal Reasoning of LLMs in Real-World Scenario
Strava for Claude Code / Codex. CLI to track your AI coding output tokens, spend, and streaks. Compete on a global leaderboard and see what your friends are building. Make every agentic coding session count.
RiskLab (ACL 2026 Demo): A Toolkit for Probing Emergent Risks in LLM-Based Multi-Agent Systems
Itemized cost receipts for Claude Code and Codex — by turn, verified, local.
Token-efficiency analyzer for AI coding agents (Claude Code, OpenAI Codex CLI): cost estimates, cache analysis, and savings advice. Zero dependencies.
The platform for developers building applications powered by LLMs
The DPAB-α Benchmark
LLM Evaluation and Observability System for Football Content
Code of our paper "A Unified Agentic Framework for Evaluating Conditional Image Generation".
Open source framework for evaluating AI Agents
First Latency-Aware Competitive LLM Agent Benchmark
Full stack observability for local AI agent development. Provides a llm-proxy to use as LLM provider base URL, logs traces to opik.
Recovery-Bench is a benchmark for evaluating the capability of LLM agents to recover from mistakes
Economical human-readable logs and structural (event pattern) retrieval for agentic trajectories
Transparent logging proxy for LLM API traffic that auto-configures clients and records every request and response to Claude, OpenAI, and other providers for debugging, auditing, and analysis.
The black box for AI agents — record every session, referee every task claim. Live mirroring + atomic coordination in a bucket YOU own. No server.
Pre-merge behavioral regression gate for AI agents. Check agent changes, inspect regressions, and block broken behavior before merge. Local-first.
A stress-testing benchmark for partial-completion evaluators in stateful multi-turn agents, focusing on temporary progress, rollback, and attribution failures.
See how sure an LLM is about each word.
OpenTelemetry tracing for DeepSeek Harness (dsh): turns each agent turn into a GenAI span tree — steps, LLM calls with TTFT, tool executions, token usage — exported over standard OTLP to Jaeger, Grafana Tempo, SigNoz, Langfuse, or any compatible backend.
⚡ Terminal HUD, hard budget breakers & prompt-cache leak alerts for AI coding agents (Claude Code, Cursor, Codex)
CSVs of the Huggingface and LMArena LLM leaderboards, along with the code to generate them in Python and R.
Arrakis is a library to conduct, track and visualize mechanistic interpretability experiments.
Real-time agent observability dashboard. View beads tasks, agent activity, and system health in TUI or React web interface. Go + React.
Local monitor for Claude Code API traffic, logs, prompts, and tools
AgentWatch — Real-time reasoning auditor and observability platform for AI agents. Catches silent failures in your agent's reasoning chain before they execute — not after they break production.
Quota monitoring for the Pi coding agent. Anthropic, Codex & Copilot and OpenRouter usage at a glance
Audit any agent decision across its past, present, and future, on one typed graph.
Blind A/B testing for AI agent skills, .cursorrules, CLAUDE.md, and system prompts. Detects placebo and harmful instructions.
Your AI coding agent just billed you. Here's the receipt. Cost receipts for Claude Code, Codex, Cursor, Gemini & opencode — no account, no upload.
[EMNLP2026] Agent Skill Evaluation and Evolution: Frameworks and Benchmarks
Self-hosted OpenAI-compatible mock server — test your LLM integrations without exposing your API key. By Axium.
The reliability & architecture analyzer for LLM and agentic systems. By ActaClad.
MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents
Observability for AI and LLM calls in Laravel. Automatically tracks requests to OpenAI, Anthropic, Gemini, Mistral and more, calculates token and media costs, enforces per-user budgets, and ships a real-time dashboard with OpenTelemetry export.
Catch AI agents that say "done" when the task isn't. Checks agent-trace/v1 traces with claim-evidence rules, LLM judges and a trained classifier, returns verified / false_success / unverifiable with calibrated confidence, and ships a 300-trace labelled benchmark.
Open-source observability for AI applications: every LLM call, prompt, token and agent decision.
A simple prompt testing library for LLMs.
Laravel AI Guard 🛡️💰🤖 - Control and optimize AI costs in Laravel AI SDK applications 🚀 Track OpenAI & LLM token usage 📊, estimate AI costs before execution ⚠️, enforce AI budgets 🧾, and prevent unexpected billing spikes in production 💥.
The open-source, local-first alternative to LangSmith. No cloud. No setup.
Postman for AI - design, evaluate, and debug LLM interactions with full transparency.
Multi-backend observability CLI for metrics, logs, and traces — unified interface, AI-Agent friendly
gradio region:us
gradio region:us
gradio region:us
Open-source cost metering for n8n AI workflows — token usage per workflow, priced daily, on your own machine.
Practical evaluation techniques, benchmarks, and examples for production AI agents: RAG, tool use, planning, memory, safety, reliability, and cost.
Real-time agent visualization in pixel office for Claude CLI and Claude macOS app with basic Codex support /// Real time визуализация ИИ агентов в виде пиксельного офиса для Claude CLI и Claude macOS app с базовой поддержкой Codex.
MASRubric: Auditing Information Flow in Multi-Agent Systems with Failure-Distilled Pitfall Rubrics.
Dashboard that allows the user to visualize in a clear and intuitive way how the OpenAI models are being used.
A calculator to estimate the number of tokens and cost associated with processing images using Azure OpenAI's multimodal models, including GPT-5.x, o-series, and image generation models