Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 31 of 57

agentledger-otel

OpenTelemetry/OTLP trace export adapter package for AgentLedger

agentledger-postgres

Postgres StateStore adapter package for AgentLedger

agentledger-py

Python SDK for AgentLedger - AI agent observability and budget management

agentledger-runtime

Runtime reliability layer for Agent Harness stacks

agentlens-io

Audit logging for Claude AI agents — transparent, tamper-evident, OSS

agentlens-kk

AgentLens Python SDK — fleet governance for multi-framework agent stacks

agentlens-monitor

LLM observability, quality scoring, hallucination detection, cost tracking, agent run debugging, and drop-in LangChain + LlamaIndex callbacks — in 2 lines of code.

agentlens-observe

Debug AI agents visually — self-hosted, open-source observability SDK

agentlens-proxy

Profile AI agents by intercepting LLM API traffic through a local MITM proxy

agentlens-sdk

Python SDK for AgentLens — observability for AI agents

agentlens-server

AgentLens observability server for AI agents

agentlens-tracer

The interactive debugger for LLM agents. Pause, inspect, and fork any agent mid-run.

agentlens-tracker

Track and debug AI agents step-by-step

agentlens-xray

Local-first AI agent debugger with fork & replay

agentlensai

Python SDK for AgentLens — observability and audit trail for AI agents

agentlenscn

See what your agents think. Open-source AI Agent observability platform.

agentlinter

Estimate costs and lint agent workflow YAML files

agentlog-llm

Trace AI agent execution — LLM calls, tool calls, costs and latency

agentlog-sdk

Python SDK for agentlog - a local-first decision log daemon for agentic workflows

agentlogger

Simple, colorful debug logs and logfiles.

agentlogs-sdk

Python SDK for AgentLogs — open-source observability for AI agents

agently-devtools

Observation and developer tooling companion package for Agently

agentmark-sdk

AgentMark SDK for Python - Tracing and Observability

agentmesh-ai

Open-source, self-hosted observability for AI agents: OTLP ingestion, tracing SDK, sessions, cost analytics, datasets and LLM-as-judge experiments, alerts, replay, and an MCP server. SQLite or PostgreSQL.

agentmesh-observability

Production observability for Agent OS - OpenTelemetry traces, Prometheus metrics, Grafana dashboards

agentmesh_drift

Mathematical drift detection library for calculating drift/hallucination scores between outputs

agentmeter

Python SDK for AgentMeter usage tracking and billing

agentmeter-caleb12

Agent execution tracing and economic event emission.

agentmeter-sdk

Track LLM costs per user. Open source SDK for AgentMeter.

agentmetering

Know what your AI agents actually cost. Track LLM tokens, tool API calls, and compute costs.

agentmetrics

Monitor your AI agents. Track cost, latency, and errors in two lines of code.

agentmirror

Production evaluation for multi-agent AI systems without ground truth.

agentmolt

Agent Control Panel — monitor, control, and kill-switch your AI agents

agentmon

AI Agent Monitoring SDK

agentmonitor

Production-ready SDK for monitoring AI agents

agentneo

A powerful tracing library for monitoring and analyzing AI agents, LLM calls, and tool interactions.

agentobs

AgentOBS — reference implementation of RFC-0001 AGENTOBS, the Observability Schema Standard for Agentic AI Systems

agentobs-debug

Developer tools for inspecting and debugging AgentOBS traces

agentobs-validate

Reference validation CLI for the AgentOBS event standard

agentops

Observability and DevTool Platform for AI Agents

agentops-app
agentops-client

AI Reliability Engineering for LLM Agents - Monitor hallucinations, latency, and throughput

agentops-cockpit

Production-grade Agent Operations (AgentOps) Platform

agentops-monitor

Monitoring SDK for AI agents built with Google ADK

agentos-platform

Deterministic record, replay, and git-bisect for AI agent runs. Reproduce a broken run byte-for-byte offline, then binary search to the commit that broke it.

agentos-python

Python SDK for Agent OS — AI agent observability platform

agentpassport-cli

CLI tools for agentpassport trace viewing and key management

agentprdiff

Guard your LLM agents in CI. Snapshot tests that catch behavioral regressions when models, prompts, or vendors change.

agentprobe-framework

A testing and evaluation framework for software agents

agentproctor

The AgentProctor module is a testing and evaluation library designed for AI agents, enabling the testing of various behaviors and properties that cannot be effectively evaluated using standard regression or unit tests. This library provides a structured approach to evaluate AI responses against defined criteria, ensuring that agents behave as expected in diverse scenarios.

agentprod

Production patterns for indie AI agents — token bucket, cost-aware routing, retry, observability.

agentprofiler

Tools to profile and analyze agent behavior.

agentproof

pytest-based behavioral testing framework for AI agents

agentproofx

Static graph verification for agent workflows — LangGraph, ADK, AutoGen, CrewAI

agentpulse

AI Agent Observability SDK - Auto-instrument your AI agents with one line of code

agentpulse-ai

Lightweight observability for AI agents — track costs, tokens, and traces

agentpulse-astrik

Business intelligence observability layer for agentic AI systems

agentpulse-cli

Real-time AI Agent activity dashboard — sessions, tokens, tools, costs at a glance

agentpulse-py

Lightweight observability SDK for AI agents — non-blocking run tracking with batching and retries

agentpulse-sdk

Zero-config agent observability SDK — auto-captures LLM calls, costs, and traces

agentpulse-sdk-core

Self-healing telemetry proxy SDK for AI Agents

agentpulse-telemetry

Self-healing telemetry proxy SDK for AI Agents

agentpytest

Pytest for AI agents — record, score, replay, regress. Vendorable, judge-agnostic, domain-neutral. (Placeholder release; full library coming soon.)

agentq

AgentQ SDK — agent observability for Python.

agentqa

Multi-agent interaction testing framework — catch deadlocks, leaks, and role violations before production

agentqualify

Open-source toolkit for evaluating AI agents using AWS Bedrock AgentCore Evaluations

agentrace

Transparent tracing and instrumentation for LangGraph agents

agentrace-ai

Lightweight OpenTelemetry-native tracing for LLM and AI agent applications

agentracer

Lightweight AI incident detection. Catch cost spikes, latency anomalies, and prompt bloat before they hit your users.

agentradar

Local-first visual debugger for multi-agent AI systems

agentregress

Regression testing for AI agents. Turn production failures into automated tests.

agentreplay

Agentreplay is a high-performance observability and tracing platform for LLM agents and AI apps, combining semantic search, specialized evals (RAGAS, G-Eval, toxicity), and Git-like versioning of prompts and responses.

agentreplayx

Record, replay, and diff agent runs (LLMs + tools) for reproducibility and regression testing.

agentrial

Statistical evaluation framework for AI agents - pytest for agent trajectories

agentrx

AgentRx – Agentic trajectory analysis pipeline for invariant checking and LLM-as-a-Judge evaluation

agentscope-monitor

AgentScope - Distributed tracing and monitoring for AI agents

agentscope-otel

Python SDK for AgentScope — AI Agent Observability with OpenTelemetry

agentscope-otel-langfuse

Langfuse OpenTelemetry tracing integration for AgentScope applications

agentsentry

Production Infrastructure for AI Agents - Cost controls, observability, and framework-agnostic deployment

agentserver

A server for managing monitoring agents

agentsesh

Agent session intelligence — behavioral analysis, grading, remediation, replay, outcome testing, and prompt debugging for AI agent sessions

agentshadow

AgentTrace Python SDK

agentsight

Python SDK for AgentSight: seamlessly track AI conversations, metrics, and deliver client-facing insights and dashboards.

agentsim

AgentSim is a simulation based evaluation framework for multi-step agents.

agentsitter

The babysitter for your AI agents. Watches costs 24/7 and alerts you if something goes wrong.

agentskeptic

Python-native database-truth verification for CrewAI and LangGraph (AgentSkeptic kernel).

agentslow

Diagnose and auto-fix AI agent performance bottlenecks.

agentsnap-py

Snapshot tests for AI agents. Record an agent's tool-call trace, diff against a baseline, fail CI on regressions. Python port of @mukundakatta/agentsnap.

agentsociety-benchmark

A tool for benchmark your agent design in different scenarios.

agentsonar

Coordination intelligence for AI. Detection, prevention, governance, and FinOps across single agents, agents calling tools, MCP servers, multi-agent orchestrators, RAG pipelines, and custom buses. CrewAI, LangGraph, custom-orchestrator, and Claude Code adapters.

agentspine

Open-source adaptive control plane for production AI agents

agentstax-eval

A lightweight Python library for evaluating AI agents and RAG pipelines.

agentstracer

Review, score, and curate your coding agent conversation traces locally

agentstv

Web-based agent session visualizer for Claude Code

agentsview

Local web viewer for AI agent sessions

agenttap

Wire-level prompt introspection for LLM SDK calls. See exactly what was sent, with credentials redacted by default. Anthropic, OpenAI, any httpx-based client.

agenttel

AgentTel Python SDK — AI-native telemetry for Python services

agenttelemetry

OpenTelemetry-based observability for autonomous AI agent systems

agenttest

A pytest-like testing framework for AI agents and prompts

agenttest-py

The pytest of AI agents. Eval-driven testing for LLM applications.

More in Agent frameworks & infrastructure