Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 31 of 57
OpenTelemetry/OTLP trace export adapter package for AgentLedger
Postgres StateStore adapter package for AgentLedger
Python SDK for AgentLedger - AI agent observability and budget management
Runtime reliability layer for Agent Harness stacks
Audit logging for Claude AI agents — transparent, tamper-evident, OSS
AgentLens Python SDK — fleet governance for multi-framework agent stacks
LLM observability, quality scoring, hallucination detection, cost tracking, agent run debugging, and drop-in LangChain + LlamaIndex callbacks — in 2 lines of code.
Debug AI agents visually — self-hosted, open-source observability SDK
Profile AI agents by intercepting LLM API traffic through a local MITM proxy
Python SDK for AgentLens — observability for AI agents
AgentLens observability server for AI agents
The interactive debugger for LLM agents. Pause, inspect, and fork any agent mid-run.
Track and debug AI agents step-by-step
Local-first AI agent debugger with fork & replay
Python SDK for AgentLens — observability and audit trail for AI agents
See what your agents think. Open-source AI Agent observability platform.
Estimate costs and lint agent workflow YAML files
Trace AI agent execution — LLM calls, tool calls, costs and latency
Python SDK for agentlog - a local-first decision log daemon for agentic workflows
Simple, colorful debug logs and logfiles.
Python SDK for AgentLogs — open-source observability for AI agents
Observation and developer tooling companion package for Agently
AgentMark SDK for Python - Tracing and Observability
Open-source, self-hosted observability for AI agents: OTLP ingestion, tracing SDK, sessions, cost analytics, datasets and LLM-as-judge experiments, alerts, replay, and an MCP server. SQLite or PostgreSQL.
Production observability for Agent OS - OpenTelemetry traces, Prometheus metrics, Grafana dashboards
Mathematical drift detection library for calculating drift/hallucination scores between outputs
Python SDK for AgentMeter usage tracking and billing
Agent execution tracing and economic event emission.
Track LLM costs per user. Open source SDK for AgentMeter.
Know what your AI agents actually cost. Track LLM tokens, tool API calls, and compute costs.
Monitor your AI agents. Track cost, latency, and errors in two lines of code.
Production evaluation for multi-agent AI systems without ground truth.
Agent Control Panel — monitor, control, and kill-switch your AI agents
AI Agent Monitoring SDK
Production-ready SDK for monitoring AI agents
A powerful tracing library for monitoring and analyzing AI agents, LLM calls, and tool interactions.
AgentOBS — reference implementation of RFC-0001 AGENTOBS, the Observability Schema Standard for Agentic AI Systems
Developer tools for inspecting and debugging AgentOBS traces
Reference validation CLI for the AgentOBS event standard
Observability and DevTool Platform for AI Agents
AI Reliability Engineering for LLM Agents - Monitor hallucinations, latency, and throughput
Production-grade Agent Operations (AgentOps) Platform
Monitoring SDK for AI agents built with Google ADK
Deterministic record, replay, and git-bisect for AI agent runs. Reproduce a broken run byte-for-byte offline, then binary search to the commit that broke it.
Python SDK for Agent OS — AI agent observability platform
CLI tools for agentpassport trace viewing and key management
Guard your LLM agents in CI. Snapshot tests that catch behavioral regressions when models, prompts, or vendors change.
A testing and evaluation framework for software agents
The AgentProctor module is a testing and evaluation library designed for AI agents, enabling the testing of various behaviors and properties that cannot be effectively evaluated using standard regression or unit tests. This library provides a structured approach to evaluate AI responses against defined criteria, ensuring that agents behave as expected in diverse scenarios.
Production patterns for indie AI agents — token bucket, cost-aware routing, retry, observability.
Tools to profile and analyze agent behavior.
pytest-based behavioral testing framework for AI agents
Static graph verification for agent workflows — LangGraph, ADK, AutoGen, CrewAI
AI Agent Observability SDK - Auto-instrument your AI agents with one line of code
Lightweight observability for AI agents — track costs, tokens, and traces
Business intelligence observability layer for agentic AI systems
Real-time AI Agent activity dashboard — sessions, tokens, tools, costs at a glance
Lightweight observability SDK for AI agents — non-blocking run tracking with batching and retries
Zero-config agent observability SDK — auto-captures LLM calls, costs, and traces
Self-healing telemetry proxy SDK for AI Agents
Self-healing telemetry proxy SDK for AI Agents
Pytest for AI agents — record, score, replay, regress. Vendorable, judge-agnostic, domain-neutral. (Placeholder release; full library coming soon.)
AgentQ SDK — agent observability for Python.
Multi-agent interaction testing framework — catch deadlocks, leaks, and role violations before production
Open-source toolkit for evaluating AI agents using AWS Bedrock AgentCore Evaluations
Transparent tracing and instrumentation for LangGraph agents
Lightweight OpenTelemetry-native tracing for LLM and AI agent applications
Lightweight AI incident detection. Catch cost spikes, latency anomalies, and prompt bloat before they hit your users.
Local-first visual debugger for multi-agent AI systems
Regression testing for AI agents. Turn production failures into automated tests.
Agentreplay is a high-performance observability and tracing platform for LLM agents and AI apps, combining semantic search, specialized evals (RAGAS, G-Eval, toxicity), and Git-like versioning of prompts and responses.
Record, replay, and diff agent runs (LLMs + tools) for reproducibility and regression testing.
Statistical evaluation framework for AI agents - pytest for agent trajectories
AgentRx – Agentic trajectory analysis pipeline for invariant checking and LLM-as-a-Judge evaluation
AgentScope - Distributed tracing and monitoring for AI agents
Python SDK for AgentScope — AI Agent Observability with OpenTelemetry
Langfuse OpenTelemetry tracing integration for AgentScope applications
Production Infrastructure for AI Agents - Cost controls, observability, and framework-agnostic deployment
A server for managing monitoring agents
Agent session intelligence — behavioral analysis, grading, remediation, replay, outcome testing, and prompt debugging for AI agent sessions
AgentTrace Python SDK
Python SDK for AgentSight: seamlessly track AI conversations, metrics, and deliver client-facing insights and dashboards.
AgentSim is a simulation based evaluation framework for multi-step agents.
The babysitter for your AI agents. Watches costs 24/7 and alerts you if something goes wrong.
Python-native database-truth verification for CrewAI and LangGraph (AgentSkeptic kernel).
Diagnose and auto-fix AI agent performance bottlenecks.
Snapshot tests for AI agents. Record an agent's tool-call trace, diff against a baseline, fail CI on regressions. Python port of @mukundakatta/agentsnap.
A tool for benchmark your agent design in different scenarios.
Coordination intelligence for AI. Detection, prevention, governance, and FinOps across single agents, agents calling tools, MCP servers, multi-agent orchestrators, RAG pipelines, and custom buses. CrewAI, LangGraph, custom-orchestrator, and Claude Code adapters.
Open-source adaptive control plane for production AI agents
A lightweight Python library for evaluating AI agents and RAG pipelines.
Review, score, and curate your coding agent conversation traces locally
Web-based agent session visualizer for Claude Code
Local web viewer for AI agent sessions
Wire-level prompt introspection for LLM SDK calls. See exactly what was sent, with credentials redacted by default. Anthropic, OpenAI, any httpx-based client.
AgentTel Python SDK — AI-native telemetry for Python services
OpenTelemetry-based observability for autonomous AI agent systems
A pytest-like testing framework for AI agents and prompts
The pytest of AI agents. Eval-driven testing for LLM applications.