Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 24 of 57

traceforge★ 3

Agent runtime tracing + LLM-mock replay for Python. Self-contained HTML reports, pytest snapshot testing, cost tracking. No SaaS.

Inflion★ 3

Observability and influence tracing infrastructure for multi-agent AI systems.

gwBenchmarks★ 3

A running multi-task multi-data terminal science benchmark for LLMs in GW astronomy

chrome_extension-gpt_stats★ 3

GPT Stats is a lightweight Chrome extension that tracks ChatGPT usage effortlessly. It automatically measures the time you spend, counts your prompts, and displays useful usage statistics.

trace-weft★ 3

Rust-first, local-first observability & debugging for LLM agents: capture, inspect, replay, diff, and export traces (OpenTelemetry-compatible).

agenttrace★ 3

Trace and debug AI agent behavior locally with a step-by-step visual tool that stores data offline for clear inspection and faster development.

RAGScope★ 3

A lightweight observability tool for visualizing and comparing RAG retrieval strategies. Features real-time embedding visualization and side-by-side performance metrics.

agentic-eval-multi-participant-coordination★ 3

AI agent evaluation framework for multi-participant coordination tasks. Built with LangGraph, custom MCP tools, and LLM-as-a-Judge evaluation. MSc dissertation project (University of Edinburgh, 2025).

agentmetrics★ 3

Open-source AI agent observability. Track cost, latency, tokens, and errors for OpenClaw, Hermes, LangChain, CrewAI, LlamaIndex, OpenAI Agents, AutoGen, and Anthropic Managed Agents. Self-hosted, no cloud required.

OpenMesh★ 3

Open-source observability and control plane for AI agent ecosystems

openclaw-office★ 3

Visualize and manage AI agent collaboration in real time using an isometric virtual office with status, tools, and resource tracking.

openclaw-dashboard★ 3

A lightweight, single-file OpenClaw dashboard for real-time agent fleet monitoring — no build step, no database, no Docker.

python-sdk★ 3
burnlens★ 3

Open-source LLM FinOps proxy — track OpenAI, Anthropic (Claude), and Google Gemini costs by feature, team, and customer. Zero code changes. pip install burnlens.

assayer★ 3

Run a prompt across multiple LLMs and compare outputs side by side in the terminal.

simple-usage★ 3

macOS menu bar app to monitor Claude Code & OpenAI Codex usage and rate limits — 5-hour / weekly quotas with reset countdowns, at a glance

EvalGov★ 3

Open-source, self-hosted evaluation and governance platform for multi-agent systems

agent-optimization★ 3

Prompt, checklist, and Agent Skill for optimizing LLM agents without benchmark hacks.

ellmos-tests★ 3

B/O/E evaluation framework for SKILL.md-based LLM operating systems and agent hubs

ai-agent-pulse★ 3

Daily automated tracking of 19 AI agent frameworks. Pulse Score ranks momentum via star velocity, release freshness, commit activity, and community health.

genai_otel_instrument★ 3

GenAI OpenTelemetry Auto-Instrumentation Library A comprehensive wrapper for automatic instrumentation of LLM/GenAI applications Supports all major LLM providers and MCP (Model Context Protocol) tool calls

harness-diagnostic★ 3

System-level lint for multi-agent harnesses. Catches the 21 structural traps single-file linters miss — including the LLM-when-you-should-use-code patterns that burn tokens.

agentlens★ 3

AgentLens — Observability and Explainability for AI Agents

agentir★ 3

A compiler infrastructure for agentic trajectories — the LLVM for agent traces.

Inferscope★ 3
agent-reward-bench-leaderboard★ 3

gradio region:us

chatbot-arena-leaderboard★ 3

gradio leaderboard region:us

performance-llm-board★ 3

gradio region:us

or-bench-leaderboard★ 3

static region:us

open-lithuanian-llm-leaderboard★ 3

gradio region:us

LLM_Alignment_Evaluation★ 3

gradio region:us

LLM-FR_Leaderboard★ 3

gradio region:us

polish-llm-benchmarks★ 3

static region:us

LLM_Arena_ConcentrateAI★ 3

docker region:us

Leaderboard-Deepseek-Gemini-Grok-GPT-Qwen★ 3

gradio region:us

agent_detective★ 3

An eval framework for multi-agent systems that names the culprit: OTEL traces in, execution graph out, blame analysis on top.

ToolFailBench★ 3

Diagnostic benchmark for measuring tool-use failures in LLM agents across 1,000 tasks in finance, medicine, law, cybersecurity, and real estate.

floe-cookbook★ 3

AI Initiative Ledger — runnable examples and reference agents

evaloop★ 3

Evaluation-driven autonomous development: a harness for agent loops whose acceptance criterion is a metric, not a test suite. Held-out metrics, sealed scoring, integrity checks.

agentura★ 3

CI/CD checks for AI agents. Run evals on every pull request, catch regressions, and see what changed before you merge. Works with any agent, no SDK required. Like pytest, but for AI agents.

skillgen-eval★ 3

Reproducible evaluation of independent Agent Skill generation and downstream execution

FastAPIChat★ 3

FastAPIChat is a powerful application for evaluating large language model based chatbots.

applied-ai-engineering-lab★ 3

Local-first AI Quality Engineering platform with LLM providers, RAG, QA agents, data validation, multi-agent workflows, evaluation, observability and governance.

ai-r★ 3
who-ran-what★ 3

track your agents, skills, and tools usage - Analytics dashboard for Claude Code, Gemini CLI, Codex CLI, and OpenCode

agent-logs-starter-kit-v3★ 3

A universal, cross-platform logging system for AI agents.

probelock★ 3

A capability lockfile for local models. Catch silent tool-calling regressions in CI (deterministic, no LLM judge).

causentra★ 3

Open-source, privacy-first observability runtime for multi-agent AI systems across frameworks, providers, OpenTelemetry, and durable transports.

dpbench★ 3

DPBench: A benchmark for evaluating coordination in multi-agent LLM systems under simultaneous resource contention.

tokin★ 3

Make any agent harness token-native.

EvoGate-RSI★ 3

Evidence-gated Recursive Self-Improvement (RSI) runtime for self-improving LLM agents, self-evolving agents, agentic AI, AI safety, evaluation, lineage, and rollback.

MemLens★ 3

Memory observability for LLM agents—trace what they remember, why, and how it shapes every response.

antigravity-token-capsule★ 3

Real-time desktop floating token monitor and performance telemetry capsule for Google Antigravity (CDP sniffing, 5-segment breakdown, TTFT & tok/s)

thai-llm-leaderboard★ 3

static region:us

tiiny-bench★ 3

An independent benchmark for the Tiiny AI Pocket Lab. Measures what the box actually does: prefill scaling, sustained throughput, concurrency and reasoning cost, with NPU budget as the currency. Web app plus a self-contained HTML report. Python stdlib only.

airun-profiler★ 3

An open-source AI infrastructure profiler that connects AI workload telemetry to physical infrastructure behavior and economics.

LLM-Agent-Tracelens★ 3
promptfoo-cicd-framework★ 3

AI Testing Framework using Promptfoo for automated LLM evaluation, red teaming, and CI/CD integration with Jenkins. Supports prompt validation, security testing, custom assertions, API-based model testing, and quality gates for enterprise GenAI applications.

agent-trace-analysis-platform★ 3

Agent Trace Analysis Platform (atap) — pluggable pipeline for trace representation, analysis, error classification, failure attribution and recovery for LLM agents

indicorderbench★ 3

Catches multilingual voice agents placing the wrong order. τ-bench style benchmark + CI gate for voice ordering agents in English and Hinglish: checks the committed order, not the transcript.

agent-sentinel★ 3

High-performance profiler & flight recorder for MCP and AI coding agents (Cursor, Claude Code) and MCP tools. Built in Go & React.

qed-proof-core★ 3

Proof of Agent Work (PoAW): an open protocol and self-hostable node that verifies what AI agents claim they did and issues signed, publicly checkable receipts.

Spatial-Gym★ 3

Gymnasium environment where AI agents solve SPaRC grid puzzles one move per step, with optional backtracking. Comes as a pip package, a Prime Intellect verifiers env and an OpenEnv server. The best model solves 16.0%, humans 98.0%.

rius-sdk-python★ 3

OpenTelemetry-native agent observability: trace every LLM call, tool call and agent step, with cost per call, alerts, and Claude Code / Codex / Cursor session tracing. Python SDK.

harness-db★ 3

HARNESS-DB: 1,256 LLM agent harnesses coded on 38 design dimensions with a verbatim quote behind every value; pre-registered PRISMA 2020 review; loader, explorer, MCP server

harness-eval★ 3

Did this change to the coding-agent harness make the code better or worse? Paired, repeated, isolated evaluation with a verdict you can trust.

sample-agentcore-behavioral-contract-evaluations★ 3

Run process-correctness validation as a custom code-based evaluator in Amazon Bedrock AgentCore Evaluations, deterministic behavioral contracts over the OTEL traces agents already emit, with audit evidence records.

ctxvitals★ 3

A Lighthouse for your AI agent stack: measures the real token cost of MCP servers, skills, and rules; flags security issues and duplicates; grades your setup A-F.

traindoctor-env★ 3

Executable RL environment for diagnosing and repairing distributed PyTorch training failures, with sandboxed execution and correctness-verified recovery rewards.

what-are-my-agents-doing★ 3

What Are My Agents Doing? (wamad) A pixel-art office that visualizes what your Claude Code agents are doing in real time. Runs locally.

craftbench-ue★ 3

Deterministic evaluation of AI coding agents in Unreal Engine

tokenleak★ 3

See where your Claude Code and Codex tokens go: a local, read-only audit of your own transcripts, plus lean-context rules and a read guard. The rules cut cost 32% over 80 A/B runs on Sonnet 5.

agent-failure-debugger★ 2

A deterministic pipeline that diagnoses, explains, and safely auto-fixes failures in LLM agent systems.

tracecraft★ 2
Openclaw-Bootstrapping-Benchmark★ 2

OpenClaw Bootstrapping Benchmark is an automated test suite that evaluates how well local LLM models (served via Ollama or compatible OpenAI-API endpoints) can complete OpenClaw's first-run "bootstrapping ritual".

llm-compare★ 2

LLM benchmark comparison tool

AgenticGames★ 2

A modular framework for evaluating LLM agents in zero-knowledge text environments through semantic feedback and autonomous reasoning.

Dreamofhyperparameters★ 2

Do LLM Agents Dream of Better Hyperparameters? Comparing 5 frontier AI agents as autonomous ML researchers on Karpathy's autoresearch framework. 819 experiments, 7 hypotheses tested.

werewolf-quack-lab★ 2

Real DuckDB Quack infrastructure for multi-agent Werewolf: containerized player nodes, Quack gateway federation, browser runner, and local/hosted LLM evals.

zistica-lumin★ 2

Migrated → github.com/zistica/korveo (rebranded). Archived.

agentlens★ 2

Open-source observability platform for AI agents. Trace every LLM call, tool use, and decision in real-time. Supports OpenAI, Anthropic, Gemini, LangChain, CrewAI, LiteLLM, MCP. Self-hosted. Zero dependencies.

CrewAI-AgentOps★ 2

CrewAI AgentOps: Monitor your AI Agents

wauldo-leaderboard★ 2

RAG Framework Leaderboard — 6 frameworks, 70 adversarial tests, weekly auto-refresh. Live at wauldo.com/leaderboard

Agora-ai-agent-visualizer★ 2

desktop app for watching claude code multi-agent debates in real time. also runs debates between any combination of models.

veritas★ 2

Multi-agent verification for AI outputs: claim verification, RAG diagnostics, pre-action verification for agentic AI. Includes ablation studies proving multi-agent vs single-prompt tradeoffs, FaithBench benchmarks, and bias-triggering evaluation methodology

llm-coordination-harness★ 2

Reproducible evaluation harness for hidden coordination variables in multi-agent LLM systems.

vigil★ 2

OpenTelemetry-native observability — GenAI semantic conventions, contract-derived spans, multi-platform (Langfuse, LangSmith, Jaeger). Part of the Life Agent OS.

support-fte-evals★ 2

Eval-driven Customer Support FTE using OpenAI Agents SDK. Multi-agent routing, guardrails, and systematic quality evaluation.

gauntlet★ 2

Production prompt regression testing for agentic flows

Lumino★ 2
evallab★ 2

A small web app for evaluating LLM outputs systematically. Define test sets, define agents, run them against an LLM-as-judge, compare the diff. Ship prompts with numbers, not vibes.

agent-prod★ 2

Production AI agent quality gate and risk control framework for LLMOps, agent evaluation, regression detection, gray release, audit, and observability

open-agent-eval★ 2

Lightweight open-source toolkit for evaluating tool-calling AI agents on safety, correctness, and reliability

ariadne-eval★ 2

Trajectory-level observability and evaluation for LLM agents. Trace, score, and detect regressions in multi-step tool-using agent runs.

semanticembed-sdk★ 2

6D structural risk analysis for directed graphs: find SPOFs and cascade risks in AI-agent pipelines, microservices, and CI/CD from topology alone. Source-available, free tier, no signup. Sub-millisecond.

agent-harness-lab★ 2

Goal-driven harness experiments on real agent runtimes — a local, stdlib-only CLI workbench with Copilot + Auto modes and evidence-first reporting.

agent-arena★ 2

Compete with AI agents in a UFC-style coding battle, solving tasks live while judging solutions on correctness, quality, and style in a local app.

debrief★ 2

A local UI for inspecting Claude Code session logs. Useful when developing subagents and skills

InferHarness★ 2

A local-first harness for testing and analysing LLM inference results.

tellme★ 2

CLI tool to verify OpenAI-compatible model deployments — check tool-call and reasoning support

More in Agent frameworks & infrastructure