Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 51 of 57
Regression tests and explainable release gates for RAG and AI agents
A lightweight evaluation and regression detection library for RAG pipelines.
Respan instrumentation plugin for AgentOps
Respan instrumentation plugin for AgentScope
Respan instrumentation plugin for Microsoft Agent Framework
Respan instrumentation plugin for Ragas evaluations and experiments
Respan instrumentation plugin for Microsoft Semantic Kernel
Record-replay debugger for AI agent runs
Local agent run receipts with signed JSONL records and Markdown review artifacts.
TTTPS Proof-of-Time step callback for smolagents: cryptographic audit-trail timestamps for agent step outputs via the public KPP Provenance API
OpenTelemetry AWS Bedrock Runtime instrumentation
OpenTelemetry Bedrock AgentCore instrumentation
AI Agent observability SDK and XAgent adapter
Load- and chaos-testing framework for AI agent orchestration plumbing
TAPM v2 hook for Anthropic (in-process library instrumentation)
TAPM v2 hook for CrewAI (in-process library instrumentation)
TAPM v2 hook for LiteLLM (in-process library instrumentation)
TAPM v2 hook for OpenAI Agents SDK (in-process library instrumentation)
Anthropic integration for telemetry.dev Python SDK
AWS Bedrock integration for telemetry.dev Python SDK
LiteLLM integration for telemetry.dev Python SDK
OpenAI integration for telemetry.dev Python SDK
An open-source improvement loop for production AI agents.
Library to systematically track and evaluate LLM based applications.
TTTPS Proof-of-Time ASGI middleware for vLLM: cryptographic audit-trail timestamps for OpenAI-compatible completions via the public KPP Provenance API
Harness-agnostic adapter that traces coding-agent sessions to W&B Weave
Framework-agnostic conformance suite for AI agent implementations
AI Agent node (baseado no n8n-nodes-langchain.agent v2.21.0) com tracing para Langfuse
Local HTTPS interception proxy — records desktop AI agent calls (Claude Desktop, Cursor, etc.) to ARKNA
Track AI agent interactions (MCP tool calls and more) with Keoki analytics.
Zero-dependency OTLP span contract for agent traces: span shape, gen_ai attribute vocabulary, pure builders, and a conformance validator that reports what a trace can and cannot answer
Usage tracking, credit management, execution-trace collection, and telemetry export for the Agent Dev Container SDK.
AgentTap — agent-native, local-first LLM observability in one process: four capture channels (OTLP traces/logs ingest, built-in LLM tracing proxy, Claude Code transcript import with input reconstruction), SQLite store, and a LangSmith-style three-pane das
Run a prompt through a LangChain (system + human) chain over Gemini on Vertex AI; optional LangSmith tracing.
NestJS/LangChain integration for the Langfuse v5 OpenTelemetry SDK: a noise-filtered LangChain callback handler, a version-stepped Langfuse-managed prompt runnable, and helpers for tracing detached session turns.
Drop-in LangChain + LangGraph instrumentation with EvalGuard guardrails, trace logging & cost tracking
Countly AI observability adapter for LangChain
TypeScript runner for Sovara — wrap a script in a single function to record every LLM call, tool invocation, and log line into a structured run graph.
Statsig helps you move faster with feature gates (feature flags), and/or dynamic configs. It also allows you to run A/B/n tests to validate your new features and understand their impact on your KPIs. If you're new to Statsig, check out our product and cre
A dead man's switch for autonomous agent fleets. Finds the agents that went quiet, and the ones that lie about succeeding.
Collector CLI for Burn — syncs multi-agent usage aggregates. Aggregates only: never prompts, paths, or content.
Turn reviewed MCP failures into deterministic recovery evidence and executable regression tests.
A self-hostable notification layer for AI agents — they report what they did and what needs you; you read one feed and click out to where the work lives.
Artifact contracts, silent-failure detection, and LLM autopsy reports for unattended AI agents
Execution traces for agent behavior - OpenTelemetry-style observability for AI agent workflows
Behavior analytics service for agent-exec-trace - transforms raw traces into operator-ready signals
Read API service for agent-exec-trace - serves product-facing views from the normalized Postgres read model
An LLM that reads your agent's error and decision logs and drafts root-cause lessons
Canonical session-journal format, replay engine, and diff tool for AI agent transcripts. Zero-dependency dual-package (Python + Node.js).
A performance review for your AI agent: one CLI, one YAML test file, one markdown report.
Detect loops and wasted calls in AI agents
Observability for Claude Code subagents. Parses session transcripts, surfaces what agents actually did, and flags the failure patterns.
Pytest-style behavioral testing for AI agent pipelines
Benchmark and compare AI agents with deterministic, reproducible tests.
Interactive TUI for debugging token usage of Claude Code sessions
Dynatrace OneAgent Python code module for Azure Functions
Placeholder to reserve this name — not the real package.
LiteLLM proxy logging callback that writes request telemetry to DuckLake directly, or through a Quack or Arrow Flight SQL server
Local collector for LLM token usage emitted through OpenTelemetry
Local-first telemetry and tamper-evident receipts for AI-agent runs
Moesif observability plugin for LiteLLM
AI agent activity monitor — Python port of github.com/perplexityai/numbat
OpenInference Cohere Instrumentation
OpenInference Ollama Instrumentation
Regression testing and configuration sweeps for RAG pipelines.
TTTPS Proof-of-Time provenance metric for Ragas: verify that a response carries a genuine audit-trail timestamp from the public KPP Provenance API
Benchmark chunking and retrieval strategies on your own data: compare cost, latency, and accuracy across chunker x retriever combinations.
Reproducible agentic workflows with observability and online experimentation
Plain-English observability for AI agents — live dashboards, replay, and one-line LangGraph auto-instrumentation, metadata-safe by default.
Warpscale vLLM telemetry collector
Lynx trace analysis AI agent
Self-hostable API, viewer, and CLI for sharing AI agent conversation threads
Parasitic AI Agent Token simulator — writes mock usage into ~/.claude and ~/.codex with real JSONL schemas.
Unified observability for AI agents and infrastructure. One SDK.
Statsig helps you move faster with feature gates (feature flags), and/or dynamic configs. It also allows you to run A/B/n tests to validate your new features and understand their impact on your KPIs. If you're new to Statsig, check out our product and cre
Statsig helps you move faster with feature gates (feature flags), and/or dynamic configs. It also allows you to run A/B/n tests to validate your new features and understand their impact on your KPIs. If you're new to Statsig, check out our product and cre
Chaos engineering for autonomous AI agents
Eval framework for agent skills, across Claude Code, Codex CLI, Copilot CLI and OpenCode
面向学习者的 AI Agent 执行调试器
Chaos engineering for LLM agents -- assume every agent is going to fail, and prove it.
Companion intelligence and real-time decision supervision for autonomous AI agents.
AutoGen tool integration for Attestify — governed AI loop execution, receipts, audit trails, and Attestify Trust (no-wallet agent identity + signed evidence)
CrewAI tool integration for Attestify — governed AI loop execution, receipts, audit trails, and Attestify Trust (no-wallet agent identity + signed evidence)
LangChain toolkit for Attestify — governed AI loop execution, receipts, audit trails, and Attestify Trust (no-wallet agent identity + signed evidence)
LangGraph node integration for Attestify — governed AI loop execution, receipts, audit trails, and Attestify Trust (no-wallet agent identity + signed evidence)
Local-first cost observability for Claude Code sessions
OpenAI Agents SDK recording adapter for Kitaru.
PydanticAI recording and replay adapter for Kitaru.
Endpoint-based evaluation harness for legal RAG systems
nvtop for llama.cpp: an interactive terminal dashboard for llama-server inference monitoring
Zero-dependency LLM call tracer — patch once, trace everything
A local, private anthology of every AI coding session — browse, search, spawn-tree, faithfully export and safely manage ChatGPT / Claude / Gemini / Codex history. No network, no egress.
LLMTivo - record real LLM calls once, replay them forever, on the filesystem
LangChain audit callback integration for MemStrata Governance
An open-source, pip-installable Python library that quantitatively evaluates RAG pipelines using LLM-as-a-Judge methodology.
Monitor, debug, and optimize your AI agents. Track LLM calls, costs, latency, and tool usage in real-time.
See exactly what your AI agent did. Lightweight tracing + a local dashboard for debugging LangChain agent runs.
ThinkHive SDK v4.0 - AI agent observability with business metrics, ROI analytics, and 25+ trace format support
Zero-config agentegrity adapter for the OpenAI Agents JS SDK (@openai/agents). Mirrors the Python agentegrity.openai_agents module.
Statsig helps you move faster with feature gates (feature flags), and/or dynamic configs. It also allows you to run A/B/n tests to validate your new features and understand their impact on your KPIs. If you're new to Statsig, check out our product and cre