Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 29 of 57

agent-deepeval

Local-first Agent batch evaluation and failure analysis CLI

agent-dev-cli

AI Toolkit - CLI and SDK for agent debugging and workflow visualization

agent-devtools

Chrome DevTools for AI Agents - Real-time debugging, pause, inspect, and step through your AI agent execution

agent-diagnostics

Agent Reliability Observatory — a behavioral taxonomy and annotation framework for analyzing why coding agents succeed or fail.

agent-diff

Python SDK for Agent Diff - test AI agents and train models against replicas of services

agent-doctor

Python SDK for agent-doctor — instrument AI agents with one init() call

agent-energy-budget

Agent cost control, energy budget management, and token tracking

agent-estimate

Know what an AI task will cost before you run it

agent-eval

Agent evaluation toolkit

agent-eval-cli

Open-source multi-turn AI agent simulation and evaluation

agent-eval-harness

A local-first, lightweight harness for AI agent evaluations

agent-eval-lite

Lightweight agent evaluation framework — zero dependencies, local-first, framework-agnostic

agent-eval-rpc

Python RPC client, official optimizer bridge, and DSPy metric adapter for @tangle-network/agent-eval.

agent-evaluation

A generative AI-powered framework for testing virtual agents.

agent-evaluator

Production-ready evaluation framework for AI agents — 58 metrics (25 native + 33 Harness Config) across 7 evaluation gates: goal achievement, behavioral integrity, reliability, performance, security, multi-agent coordination, and observability

agent-evidence

Evidence capture primitives for autonomous agent systems.

agent-failure-debugger

Diagnose why your LLM agent failed. Deterministic causal analysis with fix generation.

agent-flight-recorder

Debug AI Agents without burning money

agent-forensics

Black box for AI agents — capture decisions, generate forensic reports for EU AI Act compliance

agent-framework-devui

Debug UI for Microsoft Agent Framework with OpenAI-compatible API server.

agent-framework-devui-extended

Debug UI for Microsoft Agent Framework with OpenAI-compatible API server with Visualizations.

agent-framework-devui-visualizations

Debug UI for Microsoft Agent Framework with OpenAI-compatible API server with Visualizations.

agent-framework-eval

coming soon

agent-fuse

A Circuit Breaker for AI Agents - Fail Safe by Default

agent-gates

Quality gates for AI agent output validation

agent-gauge

An operational monitoring library for Crew AI applications.

agent-genesis

Agent Genesis evaluation SDK.

agent-governance-telemetry

Framework-agnostic agent governance and telemetry library

agent-guard

An operational monitoring library for Crew AI applications.

agent-harness

AI Maintainer Agent Harness for our benchmarking and Marketplace API and platform

agent-harness-cli

A lightweight scaffold for evaluating and controlling agentic tasks with structured checks and workflow state.

agent-harnesses

Harness infrastructure for running and evaluating AI agents.

agent-healthkit

Platform-agnostic health monitoring SDK for AI agents

agent-heartbeat

Dead man's switch for AI agents. Miss a heartbeat, get an alert.

agent-in-the-loop

Client library for Agent In The Loop confidence evaluation API

agent-insights

Deep insights into every AI agent session

agent-inspector

CLI launcher for Cylestio Perimeter with default OpenAI and Anthropic configs

agent-instrumentation

Automatic instrumentation for Python applications using Traceloop SDK

agent-kitchen

Dashboard for monitoring AI coding agent sessions across Claude Code and Codex CLI

agent-ledger

Idempotency and audit ledger for AI agent tool calls

agent-ledger-sdk

Official Python client for the Agent Ledger API

agent-lighthouse

Multi-Agent Observability SDK for Agent Lighthouse

agent-loop-breaker-py

Detect repeated agent steps and stop runaway loops. Python port of @mukundakatta/agent-loop-breaker.

agent-loop-detector

Detect and debug loops in AI agent execution

agent-loop-guard

Framework-agnostic agent loop detection — sliding window similarity scoring to catch stuck agents

agent-mont

A comprehensive monitoring solution for AI agent frameworks including Langchain, CrewAI, and Agno. AgentMont provides enhanced metrics and deep observability for production AI agent applications.

agent-obs-sdk

Lightweight Python SDK for Agent Observability Tracing — instruments multi-agent AI workflows

agent-observability

Agent observability with cost tracking and compliance audit trails

agent-observability-autogpt

AutoGPT plugin for Agent Observability - automatic logging, cost tracking, and compliance

agent-observability-crewai

CrewAI integration for Agent Observability - multi-agent logging, cost tracking, and compliance

agent-observability-langchain

LangChain integration for Agent Observability - structured logging, cost tracking, and compliance for AI agents

agent-observability-studio

Real-time monitoring and debugging platform for multi-agent AI systems

agent-observatory

Observability primitives for AI agents (tracing and streaming introspection)

agent-observe

Framework-agnostic observability, audit, and eval for AI agent applications

agent-ops-cli

Agent Ops (AO) - high-performance JSONL issue/event store

agent-os-observability

Production observability for Agent OS - OpenTelemetry traces, Prometheus metrics, Grafana dashboards

agent-patrol

Runtime pathology detection for AI agents. Diagnoses loops, stalls, oscillation, drift, and silent abandonment.

agent-pilot-sdk

An SDK for LLM tracing and more.

agent-quality-inspect

Evaluation package that allows benchmarking of agentic AIs from various sources and frameworks by producing statistical results which can be compared across different use cases and datasets.

agent-recorder

Local-first CLI that records every Claude Code and Codex session into a searchable SQLite database

agent-regression-lens-py

Detect regressions between baseline and current AI agent runs. Python port of @mukundakatta/agent-regression-lens.

agent-relay-api

A small runtime to trace, replay and debug multi-tool AI agents.

agent-reporter

Agent Reporter

agent-reporting

Agent Reporting

agent-rerun

Portable reproducibility seed bundle for AI-agent steps (Python port of @p-vbordei/agent-rerun)

agent-reward-bench

Official library for AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories

agent-run-diff

Compare baseline vs current agent runs and surface regressions as structured reasons: success loss, new errors, failed tool calls, output drift, step/latency/cost bloat.

agent-safety-bench-envs

Tool environments for the Agent Safety Bench (ASB) benchmark

agent-scores
agent-simulate

Voice AI agent simulator and evaluation harness

agent-skill-compiler

Python SDK for sending agent traces to a remote Agent Skill Compiler backend.

agent-snoop

Agent observability: trace any AI agent and store runs in MongoDB (and more)

agent-sprint-testkit

AgentSprint TestKit - Professional AI agent evaluation with OpenAI Evals integration

agent-spy-nidhish

An operational monitoring library for AI agent applications

agent-stability-engine

Agent Stability Engine: reproducible stability testing primitives for autonomous agents.

agent-strace

strace for AI agents. Capture and replay every tool call, LLM request, and decision point.

agent-tally

Real-time cost tracking for AI coding agents with budget limits, kill switch, and TUI dashboard.

agent-task-planning

AI reliability framework with confidence extraction and outlier mining.

agent-tasks

Task environments and evaluation harnesses for AI agents.

agent-testbed

Add your description here

agent-torch-visualize

visualizing large population models

agent-trace-card

Generate local Markdown, HTML, and JSON review cards from agent trace JSON.

agent-trace-cli

CLI tool for tracing AI-generated code changes

agent-trace-log

LLM API 代理与日志追踪系统 - 透明观察智能体行为

agent-trace-replay

Record, replay, and debug AI agent execution traces

agent-tracer

Universal tracing library for AI agent systems with LLM integration, context management, and export capabilities

agent-trajectory-replay-py

Replay and diff AI agent event trajectories for debugging regressions. Python port of @mukundakatta/agent-trajectory-replay.

agent-transparency

A comprehensive library for tracking the input, thought process, and output of AI agents

agent-tree

A terminal-based tree visualizer for AI agent traces.

agent-triage

Rank multi-agent trace failures by severity. CLI + web dashboard + OTLP receiver, with optional LLM root-cause analysis.

agent-tribulation

Agent Tribulation — Rigorous AI Agent testing and evaluation framework.

agent-uniformity

Reference implementation for the Agent Almanac Code Uniformity benchmark

agent-usage-atlas

Generate a local dashboard from Codex CLI / Claude Code / Cursor agent logs

agent-validator

A library for validating agent outputs against schemas

agent-vcr

Record, replay, and diff MCP (Model Context Protocol) interactions — like VCR for AI agents

agent-vendor-verifier

Benchmark framework for comparing LLM agent tool-call quality across vendors.

agent-vitals

Archived research code for agent failure signals. Validation claims withdrawn; see the correction record.

agent-watch

An operational monitoring library for Crew AI applications.

agent-watchdog

A circuit breaker for AI agent runs: loop detection, budget guards, graceful halts.

agent-xray

Local-first debugging, grading, and replay tooling for AI agent traces.

More in Agent frameworks & infrastructure