Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 28 of 57

@identro/eval

AI Agent Evaluation System - Test and evaluate AI agents across frameworks

lattice-agents

Interactive dashboard that visualizes agent workflows from your project files

sideseat

Your unified workspace for building and debugging AI agents

shiprpg-agent

One line to put your AI agent on the ShipRPG leaderboard

@piiiico/agent-logger

Framework-agnostic agent action logger with optional AgentLair backend

@openclaw-agent-trace/cli

CLI tool for Agent Monitor — interactive onboarding, status, logs, and local dashboard

@firmislabs/ai-inventory

AI tech stack discovery for agents. Detect frameworks, LLM SDKs, vector stores, and model files. Machine-readable inventory for agent orchestration and inter-agent discovery.

dominion-observatory-sdk

Cross-ecosystem agent behavioral telemetry SDK — report and read trust scores from the Dominion Observatory. The only MCP scoring network that accepts agent-reported runtime data.

@pathlight/sdk

SDK for Pathlight — visual debugging and observability for AI agents

agent-visibility

Real-time debug dashboard for multi-agent AI systems — topology graph, LLM turn inspector, tool call traces, memory panel, run history, replay, and webhook alerts

@entelog/sdk

Effortless oversight for AI agents. Real-time activity feeds, risk scoring, and human-readable summaries.

@2434addy/agentledger-sdk

TypeScript SDK for AgentLedger — track, audit, and monitor your AI agents.

agent-reliability

End-to-end reliability testing for multi-step AI agent workflows. Tests what DeepEval and Langfuse can't: cascading failures across 10+ step chains.

openjck

Visual debugger for AI agent loops. Step-by-step. Locally. Zero config.

agentmetrics

Monitor your AI agents. Track cost, latency, and errors in two lines of code.

@agentlair/audit-logger

Framework-agnostic agent action logger. Logs locally by default. Connect to AgentLair for persistent, queryable audit trails.

@agenttracehq/sdk

Official Node.js SDK for AgentTraceHQ - Tamper-proof audit trails for AI agents

@ravaniroshan/agentrace

Visual debugger for AI agent loops. Step-by-step. Locally. Zero config.

clevagent

Monitor your AI agents in 2 lines of code — heartbeat watchdog, loop detection, cost tracking, auto-restart

@timemachine-sdk/sdk

Time Machine SDK for AI Agent Observability — capture, debug, and replay AI agent executions

@myaswin123/agentprobe

Break your AI agent before your customers do. Stress-testing engine for AI agents.

@aerograph/sdk

Reference SDK for emitting AeroGraph trace events from AI applications.

@fiddler-ai/langgraph

LangGraph JS instrumentation for Fiddler observability — auto-traces LangGraph applications via OpenTelemetry

@eggjs/agent-tracing

Tracing support for AI agents (LangGraph, Claude Agent SDK)

@m4trix/tracing

Lightweight tracing for LangGraph and LangChain-style applications

langgraph-cassette

Debug LangGraph/LangChain agents offline with cassette record/replay + token & cost logging.

@plott-ai/sdk-langgraph

LangGraph integration for Plott Analytics

@visibe.ai/node

AI Agent Observability — Track OpenAI, LangChain, LangGraph, Bedrock, Vercel AI, Anthropic

agenttrace-langgraph

LangGraph JS adapter for agenttrace-react. Visualise LangGraph agent execution traces, tool calls, and human-in-the-loop interrupts as a live React UI.

@reflight/langgraph

OpenTelemetry tracing for LangGraph

@graphos-io/dashboard

GraphOS local dashboard — live LangGraph visualization and trace explorer

@turingpulse/sdk-langgraph

TuringPulse SDK integration for LangGraph — observability for compiled StateGraph workflows

@respan/instrumentation-langchain

Respan instrumentation plugin for LangChain, LangGraph, and Langflow

@itayelgazar/sdk-langgraph

LangGraph integration for Compass Analytics

@contextcompany/langchain

The Context Company integration for LangChain.js and LangGraph

@pisama/sdk

Catch silent failures in your AI agents. TypeScript SDK for Pisama, the failure detector for vibe-coded agents (Cursor, Claude Code, LangGraph, Vercel AI SDK).

@handit.ai/handit-ai

High-performance execution tracing for Node.js with LangChain/LangGraph/OpenAI/Vercel AI auto-instrumentation

agent-debug

Pluggable visual debugger for LangGraph-based agents.

linforge

Embeddable workbench for LangGraph Agent apps — edit, run, debug, and monitor in one place

@prmaat/langchain

PrMaat LangChain integration. A BaseCallbackHandler that signs every LangGraph event with the agent's W3C DID before it hits LangSmith — so traces become cryptographically attributable, not just framework-self-attested. Implements PrMaat Verification Spec

langgraph-js

JavaScript-based acceptance test for LangGraph observability. Runs as an HTTP server.

elastikteams_ag_ui

Real-time AI agent progress tracking UI for web and Node.js - supports LangGraph, LangChain, and custom AI workflows

agenttrace-react

Headless React primitives for AI agent execution traces, human-in-the-loop approval gates, and agent status UI. Works with LangGraph, AG-UI, and the Vercel AI SDK.

AgentFailureDiscovery

Agent Failure Discovery - Analyze agent transcripts to discover error patterns

AgentOversight

A tool to oversee and evaluate agentic AI performance with validation and guidance

AgentTinman

Forward-Deployed Research Agent for AI system failure discovery

Comprehensive-RAG-Evaluation-Metrics

This library provides a comprehensive suite of metrics to evaluate the performance of Retrieval-Augmented Generation (RAG) systems. RAG systems, which combine information retrieval with text generation, present unique evaluation challenges beyond those found in standard language generation tasks

FullMeanAveragePrecision

Expanded version MeanAveragePrecision metric torchmetrics library

GPTCostCalculator

Tool for estimating the cost of using different OpenAI API models

LLM-SDG

Synthetic data generator for LLM evaluation

LLMBillingKit

Track net margin on every LLM API call

LLMEvaluationFramework

Enterprise-Grade Python Framework for Large Language Model Evaluation & Testing

LangRAGEval

LangRAGEval is a library for evaluating responses based on faithfulness, context recall, answer relevancy, and context relevancy.

LlamaLogsDev

Client library for Llama Logs; https://llamalogs.com

OpenLLMTelemetry

End-to-end observability with built-in security guardrails.

RAGCheckerZs

A RAG Checker Python package

RAGCheckerZsV2

A RAG Checker Python package

RAGCheckerZsV3

A RAG Checker Python package

RAGCheckerZsV4

A RAG Checker Python package

RURAGE

RURAGE (Robust Universal RAG Evaluation) is a Python library developed to speed-up evaluation of RAG systems with Correctness, Faithfulness and Relevance axes using a variety of deterministic and model-based metrics.

SkillMetrics

A Python library for calculating and displaying the skill of model predictions against observations.

a2a-llm-tracker

This package helps you track your llm costs

aceagentaws

Logger

aceagentlogger

Logger

acenite-agent

Acenite agent: OpenTelemetry bootstrap + heartbeat logic

adaline-crewai

CrewAI integration for the Adaline SDK

adaline-dspy

DSPy integration for the Adaline SDK

adaline-langchain

LangChain integration for the Adaline SDK

adaline-langgraph

LangGraph integration for the Adaline SDK

adaline-litellm

LiteLLM integration for the Adaline SDK

adaline-llama-index

LlamaIndex integration for the Adaline SDK

adaline-pydantic-ai

Pydantic AI integration for the Adaline SDK

adk-agent-sim

ADK Agent Simulator - Python Plugin

adk-agentic-logging

Wide Event Logging & Observability for Google ADK Agents

adk-pixel-agents

Pixel art visualization for Google ADK agents — see your multi-agent workflows come alive!

adv-rag-eval

A standalone, zero-cost RAG hallucination evaluator.

aeneas-agentwatch

AI Agent Observability SDK — Auto-instrumentation for every LLM framework

afteragent

Capture agent runs, diagnose failure patterns, and generate workflow interventions.

agent-alerts

Structured alert routing to humans and AI agents

agent-analytics-mcp-server

Usage analytics and monitoring for AI agent tool calls — track, analyze and optimize agent behavior

agent-annotators

Reusable annotators library for agent and tool execution traces

agent-atm

Agent Token Manager (agent-atm) - A lightweight, privacy-first LLM token usage metering & quota capping SDK.

agent-autopsy-sdk

Debug AI agent failures with minimal-config execution traces

agent-belt

Evaluation harness for real headless CLI agents - reproducible multi-turn scenarios, rule + LLM scoring, cross-agent comparison

agent-breaker

Chaos Monkey for AI agents

agent-carbon

Carbon emissions tracking for LLM agent frameworks

agent-chaos

Chaos Engineering for AI Agents

agent-consistency

Reliability layer for AI agent workflows: validate state, handoffs, and outcomes before agents continue.

agent-control-evaluator-budget

Budget evaluator for agent-control -- cumulative LLM cost and token tracking

agent-control-evaluator-cisco

Cisco AI Defense evaluator for agent-control

agent-control-evaluator-galileo

Galileo evaluators (Luna, Decision Models, LLM-as-judge) for agent-control

agent-control-evaluators

Builtin evaluators for agent-control

agent-convergence-scorer

Score how similar N agent outputs are — exact match, Jaccard token overlap, divergence point, composite 0-1 score. Stdlib-only.

agent-cost-attribution-layer

Middleware proxy for AI agent request attribution and compliance audit trails

agent-cost-guardrails

Budget limits and cost guardrails for AI agent frameworks (CrewAI, AutoGen, LangGraph)

agent-cove

Web portal for monitoring and interacting with AI agent sessions

agent-cover

Framework-agnostic coverage testing and observability tool for LLM Agents.

agent-dashboard

Data model and renderer for agent workspace screens.

agent-debug

Diagnose why your AI agent failed. Root cause analysis + fix suggestions.

agent-debugger

Agent Debugger for LangChain/LangGraph

More in Agent frameworks & infrastructure