Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 30 of 57

agentanvil

Contract-based testing framework for LLM agents — hybrid metrics (objective + LLM-as-judge + human), multi-agent and A2A protocol support, deterministic record/replay envelope.

agentape

Record/replay testing for LLM agents - VCR.py but for LLM SDKs

agentarena

Benchmark AI coding agents on your own codebase

agentargus

Framework-agnostic evaluation, observability, and reliability for LLM agents.

agentautopsy

Post-mortem debugger for AI agents

agentbasis

Management & Observability SDK for AI Agents

agentbeam

Monitor, debug, and manage the cost of AI agents in production

agentbeat

Production monitoring for AI agents. Heartbeat monitoring, cost tracking, failure detection.

agentbench

A simple evaluation harness for AI agents.

agentbench-cli

Paste your agent URL. We'll tell you what's broken.

agentbenchkit
agentbill

OpenTelemetry-based SDK for tracking AI agent usage and billing

agentbill-pcu-tools-2025

LangChain callback handler for automatic usage tracking and billing with AgentBill

agentbill-py-crewai

AgentBill integration for CrewAI - Zero-config crew tracking

agentbill-py-langchain

LangChain callback handler for automatic usage tracking and billing with AgentBill

agentbill-py-sdk

OpenTelemetry-based SDK for tracking and billing AI agent usage with Cost Guard protection

agentbill-sdk

One spend ceiling per agent job, enforced before the call goes out. Bound to a task_ref, not to a calendar month.

agentbills

LLM cost intelligence + budget enforcement for AI agent fleets

agentblackbox

A flight recorder for AI agents – record every decision, tool call, and failure

agentboard

AI Agent Visualization Toolkit, AI Agents development, such as agent loop, chat messages, tools/functions, workflow and raw data types including text, dict or json, image, audio, video, etc.

agentbreak

Chaos proxy for testing LLM agent resilience — inject faults like latency, errors, and bad responses. Supports OpenAI, Anthropic, and MCP.

agentbudget

Real-time cost enforcement for AI agent sessions

agentcap

Cost guardrails for LLM agent runs.

agentcat

Analytics tool for MCP (Model Context Protocol) servers, Claude Connectors, and ChatGPT Plugins - tracks tool usage patterns and provides insights

agentcents

LLM API cost interceptor and budget enforcer for AI agents

agentchaos-sdk

Evaluate agent system robustness through controlled, runtime, non-intrusive LLM API fault injection.

agentchaos-tools

Chaos testing for agentic AI — fault injection hooks for openai-agents-python

agentcheck

Trace ⋅ Replay ⋅ Test your AI agents like real software

agentcircuit

One decorator to make any AI agent reliable. Loop detection, auto-repair, output validation, budget control.

agentclaw

Visual debugger for AI agents. See every step, every decision, every failure.

agentcloudkelp

YAML-first stress testing for AI agents. Inject faults, catch behavioral drift, enforce cost budgets.

agentcoach

Agent quality analysis and repair SDK for OpenTelemetry traces

agentcomlink

Simple chat, debug and file management panel for agents

agentcop

Universal forensic auditor for agent systems — OTel-aligned event schema, pluggable violation detectors

agentcost

Track LLM costs across OpenAI, Anthropic, Gemini, LangChain, and 3500+ models with zero code changes

agentcostin

AI Agent Cost Governance — track, control, and optimize LLM spending

agentd-sdk

Observability SDK for Claude Agent SDK - metrics, tracing, and debugging for AI agents

agentdash

Multi-Agent Systems Failure Taxonomy (MAST) annotation library using LLM-as-a-Judge

agentdb-sdk

AgentDB tells you when your agent stops behaving like itself. Causal lineage, time travel and behavioral baselines for any AI agent.

agentdbg

Deprecated: AgentDbg has been renamed to Maida. Use maida-ai instead.

agentdebugx

Portable error analysis, tracing, and recovery framework for agentic AI systems. Import as `agentdebug`.

agentdeck-ai

The open game console and Research system for investigating AI behavior

agentdisco

Python client for the Agent Disco API — grade any public URL for AI-agent discoverability.

agentdna-sdk

Sentry for AI Agents — one-line observability with local SQLite persistence

agentdog

Lightweight evaluation toolkit for AI agents — test tool use, grounding, safety, and efficiency before production

agentdrift

Detect when your AI agents get dumber. Open-source quality drift monitoring.

agentds

A comprehensive benchmarking platform for evaluating AI agent capabilities in data science tasks

agentds-bench

Python client for AgentDS-Bench: A streamlined benchmarking platform for evaluating AI agent capabilities in data science tasks

agentdx

Open-source diagnostic SDK for detecting failure pathologies in AI agent systems

agentecon

AgentEcon SDK — The Credit Score for AI Agents

agentenna

The agent awareness layer

agentenvelope

Runtime identity, telemetry quality, and cost guardrails for AI agents

agentest

Universal agent testing and evaluation toolkit - record, replay, mock, and benchmark AI agents

agenteval-ai

pytest for AI agents — catch failures before production

agenteval-py

Lightweight evaluation and observability toolkit for LLM agents

agenteval-sdk

Telemetry and Observability SDK for AI Agents — with Trace Tree MRI

agentevals

Open-source evaluators for LLM agents

agentevals-cli

Standalone framework to evaluate agent correctness based on portable OpenTelemetry traces

agentevals-evaluator-sdk

Lightweight SDK for building custom agentevals evaluators

agenteye

Command-line client for the AgentEye dashboard API

agentflowtest

A simple evaluation harness for AI agents.

agentfluent

Local-first agent analytics with prompt diagnostics

agentforge-eval-geval

LLM-judge evaluators (G-Eval) for AgentForge

agentforge-langfuse

Langfuse trace dashboard hook for AgentForge

agentforge-otel

OpenTelemetry tracing + metrics emitter for AgentForge

agentforge-testing

Richer test helpers for AgentForge (golden-set runner, snapshot, recording analysis)

agentft

Agent Flow Test (AgentFT) - A simple evaluation harness for AI agents.

agentfuzz

Chaos engineering for AI agents — inject realistic production failures (tool timeouts, malformed responses, cost spirals, prompt injection) and find out what breaks before your users do.

agentgate-eval

E2E behavioral testing for AI agents — grounded in 24 research papers. Playwright for LLM agents.

agentgear-ai

AgentGear: LLM observability, prompt management, and agentic tracing platform with SDK, FastAPI backend, and dashboard.

agentglass

Tracing and graph extraction for agents

agentgraf

Zero-infrastructure AI agent tracing for Grafana + Loki

agentguard-aegis

Python SDK for AI agent auditing and tracing

agentguard-ai-muaz-iqubal-muaz

Detect runaway AI costs before they hit production (STILL IN DEVELOPING PHASE)

agentguard-cli

Detect runaway AI costs in agentic systems before they hit production

agentguard-core-schema

Core schemas for AgentGuard forensic trace format

agentguard-llm

Production-grade fault tolerance for AI agentsidempotency, loop detection, fallback chains, async support, health monitoring, and budget enforcement for LangChain, AutoGen, CrewAI, and any LLM pipeline

agentguard-observe

MrProbe / Agent Guard customer observation SDK — ship your agent's response back to MrProbe in 6 lines.

agentguard-ram

Real-time cost observability and guardrails for AI agents

agenthog

Official Python SDK for AgentOS — observability, runtime, and security for AI agents.

agenthound

Tracks down every bug in your agent workflow.

agentic-eval-team

A multi-agent system for evaluating and refining labels on text data

agentic-evidence-sdk

Python SDK for Agentic Evidence Logger

agentic-experiments

Git-first, hypothesis-forcing experiment tracking for agent-driven ML research. Bundles a research harness for the H->E->F artifact model, uses signac for local execution/run state, and bridges to W&B for remote observability.

agentic-lab

Universal record-and-replay for LLM agents.

agentic-ledger

Runtime observability for AI agents — see exactly what your agent did, why it did it, and what it cost.

agentic-metric

Agentic coding metric monitor — multi-agent, TUI, CLI

agentic-metric-x

Local and SSH remote monitoring tool for AI coding agents — track token usage and costs across Claude Code and Codex

agentic-radar

A python library for generating reports for agentic systems.

agentic-reliability-framework

ARF AI – Proprietary, access‑controlled. This package is deprecated.

agentic-veritas

Truth engine for AI agents

agenticbudget

Lightweight cost tracking for LLM pipelines and AI agents

agentinsight

Python observability SDK for agentic systems

agentium-tracer

Python SDK for Agent Runtime Layer traces

agentixlens

AI Agent Observability — trace, monitor & debug your AI agents in real-time

agentkit-cli

Unified CLI for the Agent Quality Toolkit (agentmd, coderace, agentlint, agentreflect)

agentlab-py

Python client library for the AgentLab evaluation platform using Connect RPC

agentlantern

Devtool to document and inspect LLM agent systems.

agentledger-core

Deterministic run ledger for AI agents

agentledger-langgraph

LangGraph-compatible adapter package for AgentLedger

More in Agent frameworks & infrastructure