Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 8 of 57

ingot★ 51

Change management, SkillOpt optimization, and evidence-gated evals for agent skills.

www-project-agent-observability-standard★ 50

Agent Observability Standard

trace-agent★ 50

A java agent for tracing which can be configured via simple text file and instruments the code without rebuilding the project.

futureagi-sdk★ 50

Production-grade AI evaluation, prompt management & observability SDK. Automated evaluations with sub-100ms guardrails. No human-in-the-loop required. Python + TypeScript.

agenttop★ 50

htop for AI coding agents — monitor token usage, costs, and workflows across Claude Code, Cursor, Kiro, Codex, and Copilot

TokenSpire2★ 50

A SlayTheSpire2 Agent framework to evaluate continual learning capabilities of LLM

CO-Bench★ 50

[AAAI 2026] Benchmarking Language Model Agents in Algorithm Search for Combinatorial Optimization

agent-reward-bench★ 50

AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories

Agent-X★ 50

ICLR 2026: Agent-X Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Tasks

AgentBench★ 50

The set of AI agent model implementations, benchmarks, and others used in our paper "The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspective"

Agent-Observability-Demo★ 50

面向 AI Agent 入门者的教学项目。通过一个完整的电商客服 Agent,让你亲眼看到每一轮对话中 Agent 内部经历了哪些环节、做了什么决策、调用了什么工具、生成了什么 Prompt。

benchjack★ 50

AI agent benchmark hackability scanner — find evaluation vulnerabilities before they undermine your results

SciAgentGYM★ 50

Code for Paper: Benchmarking Multi-step Scientific Tool-use in LLM Agents

hyperweave★ 50

Visual CLI for agents. Deterministic SVG diagrams, tables, and charts with embedded structured metadata. Zero JavaScript. Renders anywhere markdown does.

llmBench★ 50

llmBench is a high-depth benchmarking tool designed to measure the raw performance of local LLM runtimes (Ollama, llama.cpp) while providing deep hardware intelligence.

opencode-sentry-monitor★ 50

Use Sentry Agent Monitoring to add observability to your Opencode

dt-evals★ 50

AI evaluators CLI for your AI apps and Agents - Dynatrace AI Observability

hermes-otel★ 50

OTel Plugin for Hermes Agent

kotlin-swe-bench★ 50

The official Kotlin benchmark by JetBrains

Broadcast★ 50

Simple and composable logging for Swift apps, servers, and coding agents.

delta-mfp-local-agents★ 50

Delta-MFP: counterfactual-replay failure diagnosis for local tool-use agents (FAGEN @ ICML 2026)

grafana-agento11y-hermes★ 50

Unofficial Grafana Agent Observability plugin for Hermes Agent

InferenceBench★ 50

Benchmarking Open-Ended Inference Optimization by AI Agents

spark-observability-skills★ 50

Open-source agent skills for debugging and optimizing Apache Spark workloads

SWE-bench-Science★ 50

SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?

aten★ 50

Always-on, cross-platform telemetry daemon for AI agent activity. Sysmon for AI agents.

PowerAgentBench★ 50

PowerAgentBench is a benchmark for evaluating power-system agents on complex, multi-step operational tasks. It provides standardized tasks, environments, and metrics to support fair, reproducible comparison across agent designs.

ai-level-check★ 50

AI 幾級了 — What level is your AI usage? Reads local agent logs and writes an LV0–LV5 report. No ranking.

mivas-bench★ 50

Multi-Industry Voice Agent Simulation Bench (MIVAS) is an indicator of voice AI performance across economic sectors. MIVAS is comprised of tasks and verifiers with granular rewards executed in production-grade multi-agent RL environments across industries where voice AI is being adopted the fastest.

paperbench-expanded★ 50

PaperBench High-Difficulty and Expanded Tasks: paper reproduction beyond machine learning. 93 tasks in 12 research areas, run in domain-native software and graded by clean replay. 12 public tasks.

dolphinbench★ 50

DolphinBench - Mapping the Pareto frontier of Agent Memory

hermes-telemetry★ 50

Budget enforcement + observability plugin for Hermes Agent. Stops runaway costs before they happen.

code★ 50

Rethinking the Evaluation of Harness Evolution for Agents

agent-eval-harness★ 50
putnambench-deepseek★ 50

An agent that solved all 672 putnambench problems

10x-bench-kit★ 50

Create your own benchmark for coding AI Agents

opencode-observability★ 49

Real-time monitoring and visualization for OpenCode agents.

EnConda-Bench★ 49

Code for paper "Process-Level Trajectory Evaluation for Environment Configuration in Software Engineering Agents"

openclaw-agent-dashboard★ 49

OpenClaw agent dashboard: glassmorphic UI to manage AI agents — task kanban, document editor, API monitoring and real-time agent tracking (HTML, CSS, JavaScript)

LOCA-bench★ 49

Benchmarking Language Agents Under Controllable and Extreme Context Growth

recursive-self-improvement-suite★ 49

A suite of open-ended, non-imitative tasks involving generalizable skills for large language model chatbots and agents to enable bootstrapped recursive self-improvement and an unambiguous AGI.

MacAgentBench★ 49

MacAgentBench: Benchmark agents where they actually work — on macOS.

beszel-agent-openwrt★ 49

📊 Pre-built Beszel Agent packages for OpenWrt

axon★ 49

Axon reads the logs your AI coding-agent harnesses already write — Claude Code, Codex, OpenCode — and shows which harness, project, agent, and model is working, how fast, at what cost, and how much code it's producing.

awesome-llm-observability★ 49

50+ curated LLM observability tools PLUS 26 Agent Skills (several with runnable, unit-tested scripts) to build, evaluate, debug, secure & monitor reliable LLM apps. Tracing, evals, guardrails, LLMOps.

euro-llm-leaderboard★ 49

gradio region:us

superpowers-bench★ 49

Can your agent use the right skills for the right tasks?

hermes-dashboard★ 49

Analytics & cost-monitoring dashboard for Hermes Agent

agent-diff★ 49

Interactive Sandboxes for AI Agent Evaluations & Reinforcement Learning on 3rd party APIs (Slack, Linear, Box, gCalendar)

harbor-index★ 49

A compact high-signal benchmark for evaluating frontier agents

seedeep★ 49

What Claude Code is doing between your prompt and its answer, drawn live from the logs it already writes: every model call, every tool, each subagent on its own context window and its own model, and what the session produced. Local and read-only — your session content never leaves the machine.

nootch★ 49

A native macOS overlay for AI provider usage and agent activity.

agent-trace-diagnostics★ 49

An Agent Runtime Cognitive Safety Diagnostic Tool based on Dynamical Systems

agentic-abstention★ 49

Official code and data for Agentic Abstention, accepted at NeurIPS 2026.

outputguard★ 48

Validate, repair, and retry LLM structured outputs. 13 repair strategies for common JSON malformations, JSON Schema validation, and retry-with-feedback prompts.

agtrace★ 48

See What Your AI Agent Is Actually Doing

instrumentation★ 48

An extensible java agent framework that instruments (modifies the bytecode at class loading time) programs running on the JVM, with the purpose of capturing method invocation events (start, finish, errors ...) and notifying custom listeners.

pinpoint-go-agent★ 48

Pinpoint Go agent

ai-agent-eval-harness★ 48

The open-source MultiAgentOps evaluation and verification harness for any industry business workflow.

pare★ 48

A research framework for evaluating proactive AI assistants through active user simulation

helm-charts★ 48

The AWS Observability Helm Charts repository contains Helm charts to provide easy mechanisms to setup the CloudWatch Agent and other collection agents to collect telemetry data such as metrics, logs and traces to send to AWS monitoring services.

clawtrace★ 48

Make your OpenClaw agents better, cheaper, and faster.

trustabl★ 48

Scan agents and auto-remediate agent reliability issues pre-production

EnterpriseClawBench★ 48

[EMNLP 2026 Industry Track] EnterpriseClawBench: Benchmarking Agents from Real Workplace Sessions

ZizkaDB★ 48

Self-hosted DB for AI agents. Prove why an agent did something , causal lineage (why()), time-travel state (at()), drift baselines. AGPL. LangChain/CrewAI/MCP.

pine-of-glass★ 48

Small observability and context management extensions for the Pi coding agent

TerminalWorld★ 48

Benchmarking Agents on Real-World Terminal Tasks

fanout★ 48

Single-binary observability platform. Ingest OTLP, store as Parquet, query with DuckDB.

agent-opentelemetry-rules★ 48

Agent OpenTelemetry (otel) rules pack: enriches opentelemetry with agent telemetry, steps, access rights and policy to explain agent failure.

SpreadsheetBench-2★ 48

[Neurips 2026] SpreadsheetBench 2: Evaluating Agents on End-to-End Business Spreadsheet Workflows

open-operator-evals★ 47

Opensource benchmark evaluating web operators/agents performance

ragrank★ 47

🎯 Your free LLM evaluation toolkit helps you assess the accuracy of facts, how well it understands context, its tone, and more. This helps you see how good your LLM applications are.

python_tracer★ 47

Lumigo's Python Distributed Tracing and Performance Agent for AWS lambda

neuro-san-benchmarking★ 47

General benchmarking apparatus for running multi-agent systems against benchmarks

AgentRails★ 47

A modern dashboard for managing and monitoring your AI agents. "Cursor for agents"

agent-battery★ 47

A macOS menu bar app that displays your AI agent usage quota as a battery indicator.

KAgent★ 47

把 Agent 的每次改文件,画成一支「股票」的 K 线

Cafe★ 47

A design-of-experiments platform for evaluating compound AI systems - find which technique drives quality, by how much, and whether the difference is real.

KimiCodeBar★ 47

专为 KimiCode 打造的用量监控小工具,在菜单中轻量化运行,一目了然!

AgentSWE★ 47

Can coding agents build the agent you actually want? A benchmark of 25 agent software engineering tasks across creation, editing and optimization.

PaperBenchX★ 47

A benchmark for scientific research agents, evaluating end-to-end paper reproduction across 12 research areas.

pde-agent-bench★ 46

PDEAgentBench: An automated benchmark framework for evaluating Code Agents on optimizing scientific PDE solvers.

mcpcat-python-sdk★ 46

MCPcat is an analytics platform for MCP server owners 🐱.

AgentBar★ 46

AgentBar is a macOS menu bar app that tracks AI coding assistant usage in one place.

langcost★ 46

LangCost is a cost intelligence and fault attribution engine for AI systems & Coding Agents to pinpoint where tokens are wasted, and what to fix first.

AgentPulse★ 46

Monitor, open, and manage Agents threads from an iPad or touch display with a macOS helper.

aws-xray-java-agent★ 46

The official AWS X-Ray Auto Instrumentation Agent for Java.

kaizen-agent★ 46

The AI Agent That Improves Your LLM App

flakestorm★ 46

Flakestorm — Automated Robustness Testing for AI Agents. Stop guessing if your agent really works. FlakeStorm generates adversarial mutations and exposes failures your manual tests and evals miss.

agentops★ 46

Notion-powered control plane for OpenClaw AI agents — monitor, dispatch, tune, backup, and clone your agent fleet

OpenLLM-Monitor★ 46

OpenLLM Monitor is a plug-and-play, real-time observability dashboard for monitoring and debugging LLM API calls across OpenAI, Ollama, OpenRouter, and more. Tracks tokens, latency, cost, retries, and lets you replay prompts — fully open-source and self-hostable.

claude-code-telemetry★ 46

Claude Code Telemetry is a lightweight bridge that captures telemetry data from Claude Code and forwards it to Langfuse for visualization, with a secure local turnkey installation under a minute.

agentgraphed★ 46

Local-first analytics dashboard for AI coding sessions. See what you built with Claude Code & Codex CLI.

leaderboard★ 46

gradio leaderboard submission:automatic test:mix judge:auto modality:text eval:generation language:Hebrew

Durable-agent-harness★ 46

A research-grade harness for evaluating long-horizon AI agents, with ablatable memory, control-plane benchmarks, deterministic simulations, and pre-registered machine-graded results.

llmff★ 46

Bounded, inspectable LLM inference pipelines from declared YAML — runs offline against Ollama or any OpenAI-compatible local server, emitting JSONL traces, inspect reports, and stable exit codes.

A2E★ 46
sample-from-prototype-to-production-agentic-ai-evaluation-and-observability-with★ 46
orrery★ 46

A local-first cockpit for your AI coding agents: Claude Code, Kimi Code, DSH, Codex, OpenCode, Z Code and Antigravity sessions in one window — accurate token accounting, disk usage, session cleanup, and a local model proxy.

agentroom★ 46

Visual pixel-art layer for AgentRoom — real-time agent monitoring with per-project offices

More in Agent frameworks & infrastructure