Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 45 of 57
Local-first deterministic profiler for agentic coding logs.
Beautiful local AI usage dashboard, model monsters, cost tracking, and exports
Production prompt regression testing for agentic flows
Official Node.js SDK for the Prometa Agentic Lifecycle Intelligence Platform
Agentic logging and application awareness. Simplified.
ScopeBlind evidence wrapper for LangChain + LangGraph. Signed receipts for tool calls, graph node transitions, and interrupt resumes.
CLI tool to visualize LangGraph OpenTelemetry traces in a browser
LangInsight callback handler for LangChain. Sends input/output traces to LangInsight in parallel on each LLM end and supports feedback (thumbs up/down) via `score()`.
Circuit breaker for AI agents — pick budget-guard, loop-killer or worth-it mode and stop runaway token spend, stuck agents or budget overruns in one wrapper. Adapters for LangChain, OpenAI Agents SDK, Claude Agent SDK, Vercel AI SDK and LangGraph Platform
Command-line interface for EvalStudio
LangChain JS instrumentation for Fiddler observability — auto-traces LangChain applications via OpenTelemetry
Language-neutral trace protocol and replay harness for production AI agents.
ZeroEval TypeScript SDK
Integration LangChain with CozeLoop | 扣子罗盘 LangChain 集成
CLI benchmark suite for evaluating AI agents across providers.
Enterprise-Grade CI/CD Quality Gate for LLM Agents
Release readiness gates and evidence for Microsoft Foundry agents
Visual stepper UI for agentrr record-and-replay
Lightning-paywalled FastAPI service — niche: OTel-Compatible Per-Task Cost Splitter for Multi-Agent Systems
Ensure your AI agents work reliably by finding and fixing issues fast.
AI Agent Cost Governance — policy engine, budget enforcement, per-tool cost tracking, multi-tenant workspaces
CLI tools, SRE observability, and sandbox isolation for the Agent Governance Toolkit
Turn Langfuse/LangSmith agent traces into human-readable Agent Activity Reports (Markdown + HTML).
Know what your agents cost. Cost intelligence for AI coding agents.
Langfuse evidence and trace export adapter package for AgentLedger
Chaos engineering for AI agents — Chaos Monkey, but for your agents and tools.
Evaluation utilities for deepagents (placeholder release).
Time-travel debugging for LangGraph agents — record, replay, fork, and catch regressions from a single SQLite file.
CLI tool that evaluates LLM outputs from production logs against a dual-dimension rubric.
Zero-config LLM API inspector — intercepts calls, shows token breakdown and cost estimates in a local dashboard
PHANTASM: Invert LLM hallucination, confabulation, and uncertainty into productive features.
Drop-in observability and trust layer for AI pipelines and agents.
Drop-in, compliance-instrumented wrapper around the official Anthropic Python SDK.
Drop-in, compliance-instrumented wrapper around the official OpenAI Python SDK.
AI-powered CLI that instruments Neatlogs observability in your project
Agent Replay Debugger MCP — record every agent step + replay deterministically. Step-debugger for agentic loops. Audit-evidence signed exports. By MEOK AI Labs.
Unit tests for agents. Define an agent with capa's capabilities.yaml, write scenarios and assertions, mock MCP servers, run in CI.
Lightweight evaluation framework that unifies inference through a single VLLM sampler and runs IF-EVAL, IFBench, WritingBench, HealthBench, Arena-Hard, and AlpacaEval end-to-end.
Local-first, CI-friendly regression testing for LLM outputs.
Accountability infrastructure for AI agents
TraceLLM — Open-source LLM observability, tracing, and replay infrastructure.
Agent evaluation framework with LLM-based grading for AI agent quality assessment
Toolkit for testing and evaluating AI agent skills
Umbrella package for recording, replaying, inspecting, and forking LLM-agent runs.
OpenTelemetry GenAI utility for standardized telemetry collection across LLM, Agent, Embedding, Tool, Retrieval, Rerank, Memory and more
Dependency-free AgentRewind core runtime for deterministic LLM-agent record, replay, and fork.
TrackJS Error Tracking agent for NodeJS
Skill Arena — benchmark skill effectiveness with controlled-variable comparison
OpenTelemetry Tracing
Eval gates with confidence intervals: deterministic gates, calibrated LLM-as-judge, agentic-trace evals, and a statistically honest regression gate for LLM pipelines
A control layer for LLM-in-the-loop reliability: wrap an unreliable generator in a verify-and-refuse feedback loop with a pluggable controller (deterministic checks, a model critic, or compositions) and composable 'circuits'. Keep one exact element in the loop and outputs are checked, corrected, or honestly refused. Zero runtime dependencies; runs with no model at all.
Connect your OpenAI agents to Oversee — two lines of code for full observability
Runtime hallucination detection middleware for RAG applications
OpenTelemetry-based trace SDK for Dockerized Python Agents.
Braintrust extension for pi. Includes automatic tracing for pi sessions, turns, LLM calls, and tool executions to Braintrust.
Langfuse extension for Pi coding agent
LLM-generated prose and YAML — realistic evaluation content so agent improvements are tested against lifelike data.
Raindrop integration for Vercel's Ash agent framework (OTel traces to Raindrop + Workshop)
The Universal Reasoning Ledger for Agentic Development
Local-first observability dashboard for AI agents. MCP-native. Run it with one command.
Explicit Node and browser capture helpers for Angular AI agent applications that choose to send events.
AAEP integration for LangChain agents via callback handler
LangChain callback adapter for AeroGraph — automatically record LangChain traces.
Visual debugger and observability dashboard for PydanticAI agent pipelines
Open-source reliability testing for tool-using AI agents: record, profile cost, detect regressions before production.
Run automated tests against Salesforce Agentforce agents (External + Internal Copilot) and score them into evidence — fully local, privacy-first, no API key required.
Python SDK for AgentInsight — LLM observability, tracing, evaluation, and prompt management
SparkNotes for your AI agents — log runs to AgentNotes with one pip install and env vars.
Counterfactual replay and causal attribution over LLM-agent trajectories: find which step actually caused a bad outcome by intervening on it.
Drop-in replacement for claude-agent-sdk that adds a proxied telemetry pipeline (PII redaction + guardrails) with OpenTelemetry and AgentOS sinks.
Self-hosted observability dashboard for agentic flows — every LLM turn, tool call, and failure captured
A multi-dimensional behavioral framework for evaluating LLM reasoning quality beyond accuracy
CrewAI observability, local-first: one line of setup traces crew kickoffs, agents, tasks, tools and LLM calls to a Lookspan dashboard on your own machine.
LangChain and LangGraph observability, local-first: a callback handler that traces chains, graph nodes, LLM calls and tools to a Lookspan dashboard on your own machine.
Lightweight monitoring dashboard for vLLM inference servers
Observability pipeline for AI agent monitoring - centralized logging, metrics collection, alerting, and sensitive data redaction
Track and analyze AI coding assistant usage across Claude Code, Codex, and more
Node.js SDK for Disseqt AI validation, prompt packs, and agentic tracing.
Intercept and log AI coding agent <-> API traffic for prompt-architecture analysis.
Chaos Engineering & Failure Diagnosis for AI Agents
Production-readiness platform for AI agents — inspect, improve, and stress-test before you ship.
Instrument LLM clients and emit usage events to Lago.
One-import LLM Observability SDK — traces, metrics and cost tracking via OpenTelemetry
Self-hostable LLM observability SDK — trace, cost, and hallucination monitoring in one line.
LangGraph source connector for OpenWright — capture agent decisions, tools, and human-in-the-loop approvals as evidence.
AI SDK harness adapter for vitest-evals.
Logger module for enterprise agent framework
Tracing module for enterprise agent framework
Metrics module for enterprise agent framework
Catch silent failures in your AI agents. TypeScript SDK for Pisama, the failure detector for vibe-coded agents (Cursor, Claude Code, LangGraph, Vercel AI SDK).
Agent Guard observability SDK for Python
Evaluation judges for AI voice agents (hallucination, response accuracy, intent, tool correctness, etc.)
Track the cost, usage, and ROI of your AI coding agents across every tool.
Entropy-based evaluation metrics for AI agent behavior, tools, trajectories, uncertainty reduction, and robustness.
LoongSuite mini-swe-agent instrumentation
OpenAI Agents SDK source connector for OpenWright — capture agent runs, tools, handoffs, and approvals as evidence.
Pattern-aware OpenTelemetry tracing for multi-agent LLM systems
LangChain and LangGraph observability for Tuner SDKs
Real-time terminal monitor for Claude Code token usage — with a clean centered dashboard, keyword analytics, and a one-line installer
Read-only Inspector and evidence viewer package for AgentLedger