Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 28 of 57
AI Agent Evaluation System - Test and evaluate AI agents across frameworks
Interactive dashboard that visualizes agent workflows from your project files
Your unified workspace for building and debugging AI agents
One line to put your AI agent on the ShipRPG leaderboard
Framework-agnostic agent action logger with optional AgentLair backend
CLI tool for Agent Monitor — interactive onboarding, status, logs, and local dashboard
AI tech stack discovery for agents. Detect frameworks, LLM SDKs, vector stores, and model files. Machine-readable inventory for agent orchestration and inter-agent discovery.
Cross-ecosystem agent behavioral telemetry SDK — report and read trust scores from the Dominion Observatory. The only MCP scoring network that accepts agent-reported runtime data.
SDK for Pathlight — visual debugging and observability for AI agents
Real-time debug dashboard for multi-agent AI systems — topology graph, LLM turn inspector, tool call traces, memory panel, run history, replay, and webhook alerts
Effortless oversight for AI agents. Real-time activity feeds, risk scoring, and human-readable summaries.
TypeScript SDK for AgentLedger — track, audit, and monitor your AI agents.
End-to-end reliability testing for multi-step AI agent workflows. Tests what DeepEval and Langfuse can't: cascading failures across 10+ step chains.
Visual debugger for AI agent loops. Step-by-step. Locally. Zero config.
Monitor your AI agents. Track cost, latency, and errors in two lines of code.
Framework-agnostic agent action logger. Logs locally by default. Connect to AgentLair for persistent, queryable audit trails.
Official Node.js SDK for AgentTraceHQ - Tamper-proof audit trails for AI agents
Visual debugger for AI agent loops. Step-by-step. Locally. Zero config.
Monitor your AI agents in 2 lines of code — heartbeat watchdog, loop detection, cost tracking, auto-restart
Time Machine SDK for AI Agent Observability — capture, debug, and replay AI agent executions
Break your AI agent before your customers do. Stress-testing engine for AI agents.
Reference SDK for emitting AeroGraph trace events from AI applications.
LangGraph JS instrumentation for Fiddler observability — auto-traces LangGraph applications via OpenTelemetry
Tracing support for AI agents (LangGraph, Claude Agent SDK)
Lightweight tracing for LangGraph and LangChain-style applications
Debug LangGraph/LangChain agents offline with cassette record/replay + token & cost logging.
LangGraph integration for Plott Analytics
AI Agent Observability — Track OpenAI, LangChain, LangGraph, Bedrock, Vercel AI, Anthropic
LangGraph JS adapter for agenttrace-react. Visualise LangGraph agent execution traces, tool calls, and human-in-the-loop interrupts as a live React UI.
OpenTelemetry tracing for LangGraph
GraphOS local dashboard — live LangGraph visualization and trace explorer
TuringPulse SDK integration for LangGraph — observability for compiled StateGraph workflows
Respan instrumentation plugin for LangChain, LangGraph, and Langflow
LangGraph integration for Compass Analytics
The Context Company integration for LangChain.js and LangGraph
Catch silent failures in your AI agents. TypeScript SDK for Pisama, the failure detector for vibe-coded agents (Cursor, Claude Code, LangGraph, Vercel AI SDK).
High-performance execution tracing for Node.js with LangChain/LangGraph/OpenAI/Vercel AI auto-instrumentation
Pluggable visual debugger for LangGraph-based agents.
Embeddable workbench for LangGraph Agent apps — edit, run, debug, and monitor in one place
PrMaat LangChain integration. A BaseCallbackHandler that signs every LangGraph event with the agent's W3C DID before it hits LangSmith — so traces become cryptographically attributable, not just framework-self-attested. Implements PrMaat Verification Spec
JavaScript-based acceptance test for LangGraph observability. Runs as an HTTP server.
Real-time AI agent progress tracking UI for web and Node.js - supports LangGraph, LangChain, and custom AI workflows
Headless React primitives for AI agent execution traces, human-in-the-loop approval gates, and agent status UI. Works with LangGraph, AG-UI, and the Vercel AI SDK.
Agent Failure Discovery - Analyze agent transcripts to discover error patterns
A tool to oversee and evaluate agentic AI performance with validation and guidance
Forward-Deployed Research Agent for AI system failure discovery
This library provides a comprehensive suite of metrics to evaluate the performance of Retrieval-Augmented Generation (RAG) systems. RAG systems, which combine information retrieval with text generation, present unique evaluation challenges beyond those found in standard language generation tasks
Expanded version MeanAveragePrecision metric torchmetrics library
Tool for estimating the cost of using different OpenAI API models
Synthetic data generator for LLM evaluation
Track net margin on every LLM API call
Enterprise-Grade Python Framework for Large Language Model Evaluation & Testing
LangRAGEval is a library for evaluating responses based on faithfulness, context recall, answer relevancy, and context relevancy.
Client library for Llama Logs; https://llamalogs.com
End-to-end observability with built-in security guardrails.
A RAG Checker Python package
A RAG Checker Python package
A RAG Checker Python package
A RAG Checker Python package
RURAGE (Robust Universal RAG Evaluation) is a Python library developed to speed-up evaluation of RAG systems with Correctness, Faithfulness and Relevance axes using a variety of deterministic and model-based metrics.
A Python library for calculating and displaying the skill of model predictions against observations.
This package helps you track your llm costs
Logger
Logger
Acenite agent: OpenTelemetry bootstrap + heartbeat logic
CrewAI integration for the Adaline SDK
DSPy integration for the Adaline SDK
LangChain integration for the Adaline SDK
LangGraph integration for the Adaline SDK
LiteLLM integration for the Adaline SDK
LlamaIndex integration for the Adaline SDK
Pydantic AI integration for the Adaline SDK
ADK Agent Simulator - Python Plugin
Wide Event Logging & Observability for Google ADK Agents
Pixel art visualization for Google ADK agents — see your multi-agent workflows come alive!
A standalone, zero-cost RAG hallucination evaluator.
AI Agent Observability SDK — Auto-instrumentation for every LLM framework
Capture agent runs, diagnose failure patterns, and generate workflow interventions.
Structured alert routing to humans and AI agents
Usage analytics and monitoring for AI agent tool calls — track, analyze and optimize agent behavior
Reusable annotators library for agent and tool execution traces
Agent Token Manager (agent-atm) - A lightweight, privacy-first LLM token usage metering & quota capping SDK.
Debug AI agent failures with minimal-config execution traces
Evaluation harness for real headless CLI agents - reproducible multi-turn scenarios, rule + LLM scoring, cross-agent comparison
Chaos Monkey for AI agents
Carbon emissions tracking for LLM agent frameworks
Chaos Engineering for AI Agents
Reliability layer for AI agent workflows: validate state, handoffs, and outcomes before agents continue.
Budget evaluator for agent-control -- cumulative LLM cost and token tracking
Cisco AI Defense evaluator for agent-control
Galileo evaluators (Luna, Decision Models, LLM-as-judge) for agent-control
Builtin evaluators for agent-control
Score how similar N agent outputs are — exact match, Jaccard token overlap, divergence point, composite 0-1 score. Stdlib-only.
Middleware proxy for AI agent request attribution and compliance audit trails
Budget limits and cost guardrails for AI agent frameworks (CrewAI, AutoGen, LangGraph)
Web portal for monitoring and interacting with AI agent sessions
Framework-agnostic coverage testing and observability tool for LLM Agents.
Data model and renderer for agent workspace screens.
Diagnose why your AI agent failed. Root cause analysis + fix suggestions.
Agent Debugger for LangChain/LangGraph