Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 45 of 57

agent-analyzer

Local-first deterministic profiler for agentic coding logs.

agentic-guide

Beautiful local AI usage dashboard, model monsters, cost tracking, and exports

@bertbr/gauntlet

Production prompt regression testing for agentic flows

prometa-sdk

Official Node.js SDK for the Prometa Agentic Lifecycle Intelligence Platform

auralogs-sdk

Agentic logging and application awareness. Simplified.

@scopeblind/langchain

ScopeBlind evidence wrapper for LangChain + LangGraph. Signed receipts for tool calls, graph node transitions, and interrupt resumes.

langgraph-otel-visualizer

CLI tool to visualize LangGraph OpenTelemetry traces in a browser

@langinsight/langchain

LangInsight callback handler for LangChain. Sends input/output traces to LangInsight in parallel on each LLM end and supports feedback (thumbs up/down) via `score()`.

@monetisebg/circuit-breaker

Circuit breaker for AI agents — pick budget-guard, loop-killer or worth-it mode and stop runaway token spend, stuck agents or budget overruns in one wrapper. Adapters for LangChain, OpenAI Agents SDK, Claude Agent SDK, Vercel AI SDK and LangGraph Platform

@evalstudio/cli

Command-line interface for EvalStudio

@fiddler-ai/langchain

LangChain JS instrumentation for Fiddler observability — auto-traces LangChain applications via OpenTelemetry

@anzalabidi/agentreplay

Language-neutral trace protocol and replay harness for production AI agents.

zeroeval

ZeroEval TypeScript SDK

@cozeloop/langchain

Integration LangChain with CozeLoop | 扣子罗盘 LangChain 集成

agent-builder-evals

CLI benchmark suite for evaluating AI agents across providers.

agentci-aadi

Enterprise-Grade CI/CD Quality Gate for LLM Agents

agentops-accelerator

Release readiness gates and evidence for Microsoft Foundry agents

agentrr-ui

Visual stepper UI for agentrr record-and-replay

milo-agent-trace-cost-attributor

Lightning-paywalled FastAPI service — niche: OTel-Compatible Per-Task Cost Splitter for Multi-Agent Systems

LangSmith

Ensure your AI agents work reliably by finding and fixing issues fast.

agent-gov-saas

AI Agent Cost Governance — policy engine, budget enforcement, per-tool cost tracking, multi-tenant workspaces

agent-governance-toolkit-cli

CLI tools, SRE observability, and sandbox isolation for the Agent Governance Toolkit

agent-panorama

Turn Langfuse/LangSmith agent traces into human-readable Agent Activity Reports (Markdown + HTML).

agent-usage

Know what your agents cost. Cost intelligence for AI coding agents.

agentledger-langfuse

Langfuse evidence and trace export adapter package for AgentLedger

chaos-agents

Chaos engineering for AI agents — Chaos Monkey, but for your agents and tools.

deepagents-evals

Evaluation utilities for deepagents (placeholder release).

langgraph-replay

Time-travel debugging for LangGraph agents — record, replay, fork, and catch regressions from a single SQLite file.

llm-eval-harness

CLI tool that evaluates LLM outputs from production logs against a dual-dimension rubric.

llm-inspect

Zero-config LLM API inspector — intercepts calls, shows token breakdown and cost estimates in a local dashboard

phantasm-llm

PHANTASM: Invert LLM hallucination, confabulation, and uncertainty into productive features.

sentinel-llm

Drop-in observability and trust layer for AI pipelines and agents.

vaxelia-ai-anthropic

Drop-in, compliance-instrumented wrapper around the official Anthropic Python SDK.

vaxelia-ai-openai

Drop-in, compliance-instrumented wrapper around the official OpenAI Python SDK.

@neatlogs/wizard

AI-powered CLI that instruments Neatlogs observability in your project

agent-replay-debugger-mcp

Agent Replay Debugger MCP — record every agent step + replay deterministically. Step-debugger for agentic loops. Audit-evidence signed exports. By MEOK AI Labs.

agenteval-framework

Unit tests for agents. Define an agent with capa's capabilities.yaml, write scenarios and assertions, mock MCP servers, run in CI.

llm-eval-framework

Lightweight evaluation framework that unifies inference through a single VLLM sampler and runs IF-EVAL, IFBench, WritingBench, HealthBench, Arena-Hard, and AlpacaEval end-to-end.

llm-tripwire

Local-first, CI-friendly regression testing for LLM outputs.

proveyouragent

Accountability infrastructure for AI agents

tracellm-cli

TraceLLM — Open-source LLM observability, tracing, and replay infrastructure.

@wundr.io/agent-eval

Agent evaluation framework with LLM-based grading for AI agent quality assessment

skills-dojo

Toolkit for testing and evaluating AI agent skills

@agentrewind/sdk

Umbrella package for recording, replaying, inspecting, and forking LLM-agent runs.

@loongsuite/otel-util-genai

OpenTelemetry GenAI utility for standardized telemetry collection across LLM, Agent, Embedding, Tool, Retrieval, Rerank, Memory and more

@agentrewind/core

Dependency-free AgentRewind core runtime for deterministic LLM-agent record, replay, and fork.

trackjs-node

TrackJS Error Tracking agent for NodeJS

@lythos/skill-arena

Skill Arena — benchmark skill effectiveness with controlled-variable comparison

@opentelemetry/sdk-trace-base

OpenTelemetry Tracing

llm-evalgate

Eval gates with confidence intervals: deterministic gates, calibrated LLM-as-judge, agentic-trace evals, and a statistically honest regression gate for LLM pipelines

llm-feedback-control

A control layer for LLM-in-the-loop reliability: wrap an unreliable generator in a verify-and-refuse feedback loop with a pluggable controller (deterministic checks, a model critic, or compositions) and composable 'circuits'. Keep one exact element in the loop and outputs are checked, corrected, or honestly refused. Zero runtime dependencies; runs with no model at all.

oversee-agents

Connect your OpenAI agents to Oversee — two lines of code for full observability

raguard-py

Runtime hallucination detection middleware for RAG applications

ylg-open-trace-agent-sdk

OpenTelemetry-based trace SDK for Dockerized Python Agents.

@braintrust/pi-extension

Braintrust extension for pi. Includes automatic tracing for pi sessions, turns, LLM calls, and tool executions to Braintrust.

pi-langfuse

Langfuse extension for Pi coding agent

@forwardimpact/libsyntheticprose

LLM-generated prose and YAML — realistic evaluation content so agent improvements are tested against lifelike data.

@raindrop-ai/ash

Raindrop integration for Vercel's Ash agent framework (OTel traces to Raindrop + Workshop)

@edenkollcinaku/vibe-log

The Universal Reasoning Ledger for Agentic Development

lookspan

Local-first observability dashboard for AI agents. MCP-native. Run it with one command.

@threadplane/telemetry

Explicit Node and browser capture helpers for Angular AI agent applications that choose to send events.

aaep-langchain-producer

AAEP integration for LangChain agents via callback handler

aerograph-langchain

LangChain callback adapter for AeroGraph — automatically record LangChain traces.

aeyeagent

Visual debugger and observability dashboard for PydanticAI agent pipelines

agentchaos-reliability

Open-source reliability testing for tool-using AI agents: record, profile cost, detect regressions before production.

agentforce-probe

Run automated tests against Salesforce Agentforce agents (External + Internal Copilot) and score them into evidence — fully local, privacy-first, no API key required.

agentinsight-sdk

Python SDK for AgentInsight — LLM observability, tracing, evaluation, and prompt management

aiagentnotes

SparkNotes for your AI agents — log runs to AgentNotes with one pip install and env vars.

causal-agent-replay

Counterfactual replay and causal attribution over LLM-agent trajectories: find which step actually caused a bad outcome by intervening on it.

computer-agent-py

Drop-in replacement for claude-agent-sdk that adds a proxied telemetry pipeline (PII redaction + guardrails) with OpenTelemetry and AgentOS sinks.

llm-agent-dashboard

Self-hosted observability dashboard for agentic flows — every LLM turn, tool call, and failure captured

llm-reasoning-quality

A multi-dimensional behavioral framework for evaluating LLM reasoning quality beyond accuracy

lookspan-crewai

CrewAI observability, local-first: one line of setup traces crew kickoffs, agents, tasks, tools and LLM calls to a Lookspan dashboard on your own machine.

lookspan-langgraph

LangChain and LangGraph observability, local-first: a callback handler that traces chains, graph nodes, LLM calls and tools to a Lookspan dashboard on your own machine.

vllm-metrics-monitor

Lightweight monitoring dashboard for vLLM inference servers

@wundr.io/agent-observability

Observability pipeline for AI agent monitoring - centralized logging, metrics collection, alerting, and sensitive data redaction

@juliantanx/aiusage

Track and analyze AI coding assistant usage across Claude Code, Codex, and more

@disseqt-ai/sdk

Node.js SDK for Disseqt AI validation, prompt packs, and agentic tracing.

agent-interlude

Intercept and log AI coding agent <-> API traffic for prompt-architecture analysis.

agentchaos-core

Chaos Engineering & Failure Diagnosis for AI Agents

agentsentinel-ai

Production-readiness platform for AI agents — inspect, improve, and stress-test before you ship.

lago-agent-sdk

Instrument LLM clients and emit usage events to Lago.

llm-observer-sdk

One-import LLM Observability SDK — traces, metrics and cost tracking via OpenTelemetry

llmscope-kresnapandu

Self-hostable LLM observability SDK — trace, cost, and hallucination monitoring in one line.

openwright-langgraph

LangGraph source connector for OpenWright — capture agent decisions, tools, and human-in-the-loop approvals as evidence.

@vitest-evals/harness-ai-sdk

AI SDK harness adapter for vitest-evals.

@prysm-ai/logger

Logger module for enterprise agent framework

@prysm-ai/tracing

Tracing module for enterprise agent framework

@prysm-ai/metrics

Metrics module for enterprise agent framework

@whoopsie/sdk

Catch silent failures in your AI agents. TypeScript SDK for Pisama, the failure detector for vibe-coded agents (Cursor, Claude Code, LangGraph, Vercel AI SDK).

actaclad-agentguard

Agent Guard observability SDK for Python

agent-observability-sdk

Evaluation judges for AI voice agents (hallucination, response accuracy, intent, tool correctness, etc.)

agent-roi-tracker

Track the cost, usage, and ROI of your AI coding agents across every tool.

entropy-agent-eval

Entropy-based evaluation metrics for AI agent behavior, tools, trajectories, uncertainty reduction, and robustness.

loongsuite-instrumentation-minisweagent

LoongSuite mini-swe-agent instrumentation

openwright-openai-agents

OpenAI Agents SDK source connector for OpenWright — capture agent runs, tools, handoffs, and approvals as evidence.

pyagent-trace

Pattern-aware OpenTelemetry tracing for multi-agent LLM systems

tuner-langchain

LangChain and LangGraph observability for Tuner SDKs

wyattmcph-claude-monitor

Real-time terminal monitor for Claude Code token usage — with a clean centered dashboard, keyword analytics, and a one-line installer

agentledger-inspector

Read-only Inspector and evidence viewer package for AgentLedger

More in Agent frameworks & infrastructure