Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 29 of 57
Local-first Agent batch evaluation and failure analysis CLI
AI Toolkit - CLI and SDK for agent debugging and workflow visualization
Chrome DevTools for AI Agents - Real-time debugging, pause, inspect, and step through your AI agent execution
Agent Reliability Observatory — a behavioral taxonomy and annotation framework for analyzing why coding agents succeed or fail.
Python SDK for Agent Diff - test AI agents and train models against replicas of services
Python SDK for agent-doctor — instrument AI agents with one init() call
Agent cost control, energy budget management, and token tracking
Know what an AI task will cost before you run it
Agent evaluation toolkit
Open-source multi-turn AI agent simulation and evaluation
A local-first, lightweight harness for AI agent evaluations
Lightweight agent evaluation framework — zero dependencies, local-first, framework-agnostic
Python RPC client, official optimizer bridge, and DSPy metric adapter for @tangle-network/agent-eval.
A generative AI-powered framework for testing virtual agents.
Production-ready evaluation framework for AI agents — 58 metrics (25 native + 33 Harness Config) across 7 evaluation gates: goal achievement, behavioral integrity, reliability, performance, security, multi-agent coordination, and observability
Evidence capture primitives for autonomous agent systems.
Diagnose why your LLM agent failed. Deterministic causal analysis with fix generation.
Debug AI Agents without burning money
Black box for AI agents — capture decisions, generate forensic reports for EU AI Act compliance
Debug UI for Microsoft Agent Framework with OpenAI-compatible API server.
Debug UI for Microsoft Agent Framework with OpenAI-compatible API server with Visualizations.
Debug UI for Microsoft Agent Framework with OpenAI-compatible API server with Visualizations.
coming soon
A Circuit Breaker for AI Agents - Fail Safe by Default
Quality gates for AI agent output validation
An operational monitoring library for Crew AI applications.
Agent Genesis evaluation SDK.
Framework-agnostic agent governance and telemetry library
An operational monitoring library for Crew AI applications.
AI Maintainer Agent Harness for our benchmarking and Marketplace API and platform
A lightweight scaffold for evaluating and controlling agentic tasks with structured checks and workflow state.
Harness infrastructure for running and evaluating AI agents.
Platform-agnostic health monitoring SDK for AI agents
Dead man's switch for AI agents. Miss a heartbeat, get an alert.
Client library for Agent In The Loop confidence evaluation API
Deep insights into every AI agent session
CLI launcher for Cylestio Perimeter with default OpenAI and Anthropic configs
Automatic instrumentation for Python applications using Traceloop SDK
Dashboard for monitoring AI coding agent sessions across Claude Code and Codex CLI
Idempotency and audit ledger for AI agent tool calls
Official Python client for the Agent Ledger API
Multi-Agent Observability SDK for Agent Lighthouse
Detect repeated agent steps and stop runaway loops. Python port of @mukundakatta/agent-loop-breaker.
Detect and debug loops in AI agent execution
Framework-agnostic agent loop detection — sliding window similarity scoring to catch stuck agents
A comprehensive monitoring solution for AI agent frameworks including Langchain, CrewAI, and Agno. AgentMont provides enhanced metrics and deep observability for production AI agent applications.
Lightweight Python SDK for Agent Observability Tracing — instruments multi-agent AI workflows
Agent observability with cost tracking and compliance audit trails
AutoGPT plugin for Agent Observability - automatic logging, cost tracking, and compliance
CrewAI integration for Agent Observability - multi-agent logging, cost tracking, and compliance
LangChain integration for Agent Observability - structured logging, cost tracking, and compliance for AI agents
Real-time monitoring and debugging platform for multi-agent AI systems
Observability primitives for AI agents (tracing and streaming introspection)
Framework-agnostic observability, audit, and eval for AI agent applications
Agent Ops (AO) - high-performance JSONL issue/event store
Production observability for Agent OS - OpenTelemetry traces, Prometheus metrics, Grafana dashboards
Runtime pathology detection for AI agents. Diagnoses loops, stalls, oscillation, drift, and silent abandonment.
An SDK for LLM tracing and more.
Evaluation package that allows benchmarking of agentic AIs from various sources and frameworks by producing statistical results which can be compared across different use cases and datasets.
Local-first CLI that records every Claude Code and Codex session into a searchable SQLite database
Detect regressions between baseline and current AI agent runs. Python port of @mukundakatta/agent-regression-lens.
A small runtime to trace, replay and debug multi-tool AI agents.
Agent Reporter
Agent Reporting
Portable reproducibility seed bundle for AI-agent steps (Python port of @p-vbordei/agent-rerun)
Official library for AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories
Compare baseline vs current agent runs and surface regressions as structured reasons: success loss, new errors, failed tool calls, output drift, step/latency/cost bloat.
Tool environments for the Agent Safety Bench (ASB) benchmark
Voice AI agent simulator and evaluation harness
Python SDK for sending agent traces to a remote Agent Skill Compiler backend.
Agent observability: trace any AI agent and store runs in MongoDB (and more)
AgentSprint TestKit - Professional AI agent evaluation with OpenAI Evals integration
An operational monitoring library for AI agent applications
Agent Stability Engine: reproducible stability testing primitives for autonomous agents.
strace for AI agents. Capture and replay every tool call, LLM request, and decision point.
Real-time cost tracking for AI coding agents with budget limits, kill switch, and TUI dashboard.
AI reliability framework with confidence extraction and outlier mining.
Task environments and evaluation harnesses for AI agents.
Add your description here
visualizing large population models
Generate local Markdown, HTML, and JSON review cards from agent trace JSON.
CLI tool for tracing AI-generated code changes
LLM API 代理与日志追踪系统 - 透明观察智能体行为
Record, replay, and debug AI agent execution traces
Universal tracing library for AI agent systems with LLM integration, context management, and export capabilities
Replay and diff AI agent event trajectories for debugging regressions. Python port of @mukundakatta/agent-trajectory-replay.
A comprehensive library for tracking the input, thought process, and output of AI agents
A terminal-based tree visualizer for AI agent traces.
Rank multi-agent trace failures by severity. CLI + web dashboard + OTLP receiver, with optional LLM root-cause analysis.
Agent Tribulation — Rigorous AI Agent testing and evaluation framework.
Reference implementation for the Agent Almanac Code Uniformity benchmark
Generate a local dashboard from Codex CLI / Claude Code / Cursor agent logs
A library for validating agent outputs against schemas
Record, replay, and diff MCP (Model Context Protocol) interactions — like VCR for AI agents
Benchmark framework for comparing LLM agent tool-call quality across vendors.
Archived research code for agent failure signals. Validation claims withdrawn; see the correction record.
An operational monitoring library for Crew AI applications.
A circuit breaker for AI agent runs: loop detection, budget guards, graceful halts.
Local-first debugging, grading, and replay tooling for AI agent traces.