Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 30 of 57
Contract-based testing framework for LLM agents — hybrid metrics (objective + LLM-as-judge + human), multi-agent and A2A protocol support, deterministic record/replay envelope.
Record/replay testing for LLM agents - VCR.py but for LLM SDKs
Benchmark AI coding agents on your own codebase
Framework-agnostic evaluation, observability, and reliability for LLM agents.
Post-mortem debugger for AI agents
Management & Observability SDK for AI Agents
Monitor, debug, and manage the cost of AI agents in production
Production monitoring for AI agents. Heartbeat monitoring, cost tracking, failure detection.
A simple evaluation harness for AI agents.
Paste your agent URL. We'll tell you what's broken.
OpenTelemetry-based SDK for tracking AI agent usage and billing
LangChain callback handler for automatic usage tracking and billing with AgentBill
AgentBill integration for CrewAI - Zero-config crew tracking
LangChain callback handler for automatic usage tracking and billing with AgentBill
OpenTelemetry-based SDK for tracking and billing AI agent usage with Cost Guard protection
One spend ceiling per agent job, enforced before the call goes out. Bound to a task_ref, not to a calendar month.
LLM cost intelligence + budget enforcement for AI agent fleets
A flight recorder for AI agents – record every decision, tool call, and failure
AI Agent Visualization Toolkit, AI Agents development, such as agent loop, chat messages, tools/functions, workflow and raw data types including text, dict or json, image, audio, video, etc.
Chaos proxy for testing LLM agent resilience — inject faults like latency, errors, and bad responses. Supports OpenAI, Anthropic, and MCP.
Real-time cost enforcement for AI agent sessions
Cost guardrails for LLM agent runs.
Analytics tool for MCP (Model Context Protocol) servers, Claude Connectors, and ChatGPT Plugins - tracks tool usage patterns and provides insights
LLM API cost interceptor and budget enforcer for AI agents
Evaluate agent system robustness through controlled, runtime, non-intrusive LLM API fault injection.
Chaos testing for agentic AI — fault injection hooks for openai-agents-python
Trace ⋅ Replay ⋅ Test your AI agents like real software
One decorator to make any AI agent reliable. Loop detection, auto-repair, output validation, budget control.
Visual debugger for AI agents. See every step, every decision, every failure.
YAML-first stress testing for AI agents. Inject faults, catch behavioral drift, enforce cost budgets.
Agent quality analysis and repair SDK for OpenTelemetry traces
Simple chat, debug and file management panel for agents
Universal forensic auditor for agent systems — OTel-aligned event schema, pluggable violation detectors
Track LLM costs across OpenAI, Anthropic, Gemini, LangChain, and 3500+ models with zero code changes
AI Agent Cost Governance — track, control, and optimize LLM spending
Observability SDK for Claude Agent SDK - metrics, tracing, and debugging for AI agents
Multi-Agent Systems Failure Taxonomy (MAST) annotation library using LLM-as-a-Judge
AgentDB tells you when your agent stops behaving like itself. Causal lineage, time travel and behavioral baselines for any AI agent.
Deprecated: AgentDbg has been renamed to Maida. Use maida-ai instead.
Portable error analysis, tracing, and recovery framework for agentic AI systems. Import as `agentdebug`.
The open game console and Research system for investigating AI behavior
Python client for the Agent Disco API — grade any public URL for AI-agent discoverability.
Sentry for AI Agents — one-line observability with local SQLite persistence
Lightweight evaluation toolkit for AI agents — test tool use, grounding, safety, and efficiency before production
Detect when your AI agents get dumber. Open-source quality drift monitoring.
A comprehensive benchmarking platform for evaluating AI agent capabilities in data science tasks
Python client for AgentDS-Bench: A streamlined benchmarking platform for evaluating AI agent capabilities in data science tasks
Open-source diagnostic SDK for detecting failure pathologies in AI agent systems
AgentEcon SDK — The Credit Score for AI Agents
The agent awareness layer
Runtime identity, telemetry quality, and cost guardrails for AI agents
Universal agent testing and evaluation toolkit - record, replay, mock, and benchmark AI agents
pytest for AI agents — catch failures before production
Lightweight evaluation and observability toolkit for LLM agents
Telemetry and Observability SDK for AI Agents — with Trace Tree MRI
Open-source evaluators for LLM agents
Standalone framework to evaluate agent correctness based on portable OpenTelemetry traces
Lightweight SDK for building custom agentevals evaluators
Command-line client for the AgentEye dashboard API
A simple evaluation harness for AI agents.
Local-first agent analytics with prompt diagnostics
LLM-judge evaluators (G-Eval) for AgentForge
Langfuse trace dashboard hook for AgentForge
OpenTelemetry tracing + metrics emitter for AgentForge
Richer test helpers for AgentForge (golden-set runner, snapshot, recording analysis)
Agent Flow Test (AgentFT) - A simple evaluation harness for AI agents.
Chaos engineering for AI agents — inject realistic production failures (tool timeouts, malformed responses, cost spirals, prompt injection) and find out what breaks before your users do.
E2E behavioral testing for AI agents — grounded in 24 research papers. Playwright for LLM agents.
AgentGear: LLM observability, prompt management, and agentic tracing platform with SDK, FastAPI backend, and dashboard.
Tracing and graph extraction for agents
Zero-infrastructure AI agent tracing for Grafana + Loki
Python SDK for AI agent auditing and tracing
Detect runaway AI costs before they hit production (STILL IN DEVELOPING PHASE)
Detect runaway AI costs in agentic systems before they hit production
Core schemas for AgentGuard forensic trace format
Production-grade fault tolerance for AI agentsidempotency, loop detection, fallback chains, async support, health monitoring, and budget enforcement for LangChain, AutoGen, CrewAI, and any LLM pipeline
MrProbe / Agent Guard customer observation SDK — ship your agent's response back to MrProbe in 6 lines.
Real-time cost observability and guardrails for AI agents
Official Python SDK for AgentOS — observability, runtime, and security for AI agents.
Tracks down every bug in your agent workflow.
A multi-agent system for evaluating and refining labels on text data
Python SDK for Agentic Evidence Logger
Git-first, hypothesis-forcing experiment tracking for agent-driven ML research. Bundles a research harness for the H->E->F artifact model, uses signac for local execution/run state, and bridges to W&B for remote observability.
Universal record-and-replay for LLM agents.
Runtime observability for AI agents — see exactly what your agent did, why it did it, and what it cost.
Agentic coding metric monitor — multi-agent, TUI, CLI
Local and SSH remote monitoring tool for AI coding agents — track token usage and costs across Claude Code and Codex
A python library for generating reports for agentic systems.
ARF AI – Proprietary, access‑controlled. This package is deprecated.
Truth engine for AI agents
Lightweight cost tracking for LLM pipelines and AI agents
Python observability SDK for agentic systems
Python SDK for Agent Runtime Layer traces
AI Agent Observability — trace, monitor & debug your AI agents in real-time
Unified CLI for the Agent Quality Toolkit (agentmd, coderace, agentlint, agentreflect)
Python client library for the AgentLab evaluation platform using Connect RPC
Devtool to document and inspect LLM agent systems.
Deterministic run ledger for AI agents
LangGraph-compatible adapter package for AgentLedger