Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 51 of 57

ragops

Regression tests and explainable release gates for RAG and AI agents

ragwatch-sdk

A lightweight evaluation and regression detection library for RAG pipelines.

respan-instrumentation-agentops

Respan instrumentation plugin for AgentOps

respan-instrumentation-agentscope

Respan instrumentation plugin for AgentScope

respan-instrumentation-microsoft-agent-framework

Respan instrumentation plugin for Microsoft Agent Framework

respan-instrumentation-ragas

Respan instrumentation plugin for Ragas evaluations and experiments

respan-instrumentation-semantic-kernel

Respan instrumentation plugin for Microsoft Semantic Kernel

rewind-llm

Record-replay debugger for AI agent runs

signed-agent-receipts

Local agent run receipts with signed JSONL records and Markdown review artifacts.

smolagents-openttt

TTTPS Proof-of-Time step callback for smolagents: cryptographic audit-trail timestamps for agent step outputs via the public KPP Provenance API

splunk-otel-instrumentation-bedrock

OpenTelemetry AWS Bedrock Runtime instrumentation

splunk-otel-instrumentation-bedrock-agentcore

OpenTelemetry Bedrock AgentCore instrumentation

sthg-agent-observatory-sdk

AI Agent observability SDK and XAgent adapter

swarmbreaker

Load- and chaos-testing framework for AI agent orchestration plumbing

tapm-hook-anthropic

TAPM v2 hook for Anthropic (in-process library instrumentation)

tapm-hook-crewai

TAPM v2 hook for CrewAI (in-process library instrumentation)

tapm-hook-litellm

TAPM v2 hook for LiteLLM (in-process library instrumentation)

tapm-hook-openai-agents

TAPM v2 hook for OpenAI Agents SDK (in-process library instrumentation)

telemetry-dev-anthropic

Anthropic integration for telemetry.dev Python SDK

telemetry-dev-bedrock

AWS Bedrock integration for telemetry.dev Python SDK

telemetry-dev-litellm

LiteLLM integration for telemetry.dev Python SDK

telemetry-dev-openai

OpenAI integration for telemetry.dev Python SDK

traceport-agent

An open-source improvement loop for production AI agents.

trulens-providers-anthropic

Library to systematically track and evaluate LLM based applications.

vllm-openttt

TTTPS Proof-of-Time ASGI middleware for vLLM: cryptographic audit-trail timestamps for OpenAI-compatible completions via the public KPP Provenance API

weave-agent-adapter

Harness-agnostic adapter that traces coding-agent sessions to W&B Weave

agent-koans

Framework-agnostic conformance suite for AI agent implementations

n8n-nodes-ai-agent-langfuse-local

AI Agent node (baseado no n8n-nodes-langchain.agent v2.21.0) com tracing para Langfuse

arkna-router

Local HTTPS interception proxy — records desktop AI agent calls (Claude Desktop, Cursor, etc.) to ARKNA

@keoki/agent-tracking

Track AI agent interactions (MCP tool calls and more) with Keoki analytics.

@tangle-network/agent-trace-contract

Zero-dependency OTLP span contract for agent traces: span shape, gen_ai attribute vocabulary, pure builders, and a conformance validator that reports what a trace can and cannot answer

@tangle-network/sdk-telemetry

Usage tracking, credit management, execution-trace collection, and telemetry export for the Agent Dev Container SDK.

agenttap

AgentTap — agent-native, local-first LLM observability in one process: four capture channels (OTLP traces/logs ingest, built-in LLM tracing proxy, Claude Code transcript import with input reconstruction), SQLite store, and a LangSmith-style three-pane das

@gosce/llm-observability-orchestration-agent-langchain-community

Run a prompt through a LangChain (system + human) chain over Gemini on Vertex AI; optional LangSmith tracing.

@dltech/nestjs-langfuse

NestJS/LangChain integration for the Langfuse v5 OpenTelemetry SDK: a noise-filtered LangChain callback handler, a version-stepped Langfuse-managed prompt runnable, and helpers for tracing detached session turns.

@evalguard/langchain

Drop-in LangChain + LangGraph instrumentation with EvalGuard guardrails, trace logging & cost tracking

@countly/ai-sdk-langchain

Countly AI observability adapter for LangChain

@sovara/runner

TypeScript runner for Sovara — wrap a script in a single function to record every LLM call, tool invocation, and log line into a structured run graph.

@oai-statsig/js-client

Statsig helps you move faster with feature gates (feature flags), and/or dynamic configs. It also allows you to run A/B/n tests to validate your new features and understand their impact on your KPIs. If you're new to Statsig, check out our product and cre

deadmans-switch

A dead man's switch for autonomous agent fleets. Finds the agents that went quiet, and the ones that lie about succeeding.

joinburn

Collector CLI for Burn — syncs multi-agent usage aggregates. Aggregates only: never prompts, paths, or content.

resilireplay

Turn reviewed MCP failures into deterministic recovery evidence and executable regression tests.

agent-center-app

A self-hostable notification layer for AI agents — they report what they did and what needs you; you read one feed and click out to where the work lives.

agent-coroner

Artifact contracts, silent-failure detection, and LLM autopsy reports for unattended AI agents

agent-exec-trace

Execution traces for agent behavior - OpenTelemetry-style observability for AI agent workflows

agent-exec-trace-analytics

Behavior analytics service for agent-exec-trace - transforms raw traces into operator-ready signals

agent-exec-trace-api

Read API service for agent-exec-trace - serves product-facing views from the normalized Postgres read model

agent-log-ai1

An LLM that reads your agent's error and decision logs and drafts root-cause lessons

agent-replay-journal

Canonical session-journal format, replay engine, and diff tool for AI agent transcripts. Zero-dependency dual-package (Python + Node.js).

agent-report-card

A performance review for your AI agent: one CLI, one YAML test file, one markdown report.

agentguard-kit

Detect loops and wasted calls in AI agents

agentrace-cli

Observability for Claude Code subagents. Parses session transcripts, surfaces what agents actually did, and flags the failure patterns.

agenttest-ai

Pytest-style behavioral testing for AI agent pipelines

agentverdict

Benchmark and compare AI agents with deterministic, reproducible tests.

claude-measure-usage

Interactive TUI for debugging token usage of Claude Code sessions

dynatrace-oneagentpython

Dynatrace OneAgent Python code module for Azure Functions

fastpluggy-claude-monitor

Placeholder to reserve this name — not the real package.

litellm-ducklake-sink

LiteLLM proxy logging callback that writes request telemetry to DuckLake directly, or through a Quack or Arrow Flight SQL server

llm-ot-cost-monitor

Local collector for LLM token usage emitted through OpenTelemetry

llmwitness

Local-first telemetry and tamper-evident receipts for AI-agent runs

moesif-litellm

Moesif observability plugin for LiteLLM

numbat-agent-monitor

AI agent activity monitor — Python port of github.com/perplexityai/numbat

openinference-instrumentation-cohere

OpenInference Cohere Instrumentation

openinference-instrumentation-ollama

OpenInference Ollama Instrumentation

rag-ci

Regression testing and configuration sweeps for RAG pipelines.

ragas-openttt

TTTPS Proof-of-Time provenance metric for Ragas: verify that a response carries a genuine audit-trail timestamp from the public KPP Provenance API

ragbenchpy

Benchmark chunking and retrieval strategies on your own data: compare cost, latency, and accuracy across chunker x retriever combinations.

real-agentic-workflows

Reproducible agentic workflows with observability and online experimentation

visibleagent

Plain-English observability for AI agents — live dashboards, replay, and one-line LangGraph auto-instrumentation, metadata-safe by default.

warpscale-vllm

Warpscale vLLM telemetry collector

@lynx-js/trace-analysis-agent

Lynx trace analysis AI agent

@team-harness/threadshare

Self-hostable API, viewer, and CLI for sharing AI agent conversation threads

@aivenlab/ats-simulator

Parasitic AI Agent Token simulator — writes mock usage into ~/.claude and ~/.codex with real JSONL schemas.

@morsehq-dev/sdk

Unified observability for AI agents and infrastructure. One SDK.

@oai-statsig/web-analytics

Statsig helps you move faster with feature gates (feature flags), and/or dynamic configs. It also allows you to run A/B/n tests to validate your new features and understand their impact on your KPIs. If you're new to Statsig, check out our product and cre

@oai-statsig/session-replay

Statsig helps you move faster with feature gates (feature flags), and/or dynamic configs. It also allows you to run A/B/n tests to validate your new features and understand their impact on your KPIs. If you're new to Statsig, check out our product and cre

agent-chaos-runner

Chaos engineering for autonomous AI agents

agent-exam

Eval framework for agent skills, across Claude Code, Codex CLI, Copilot CLI and OpenCode

agenteval-debugger

面向学习者的 AI Agent 执行调试器

agentgauntlet

Chaos engineering for LLM agents -- assume every agent is going to fail, and prove it.

agentic-sidecar

Companion intelligence and real-time decision supervision for autonomous AI agents.

attestify-autogen

AutoGen tool integration for Attestify — governed AI loop execution, receipts, audit trails, and Attestify Trust (no-wallet agent identity + signed evidence)

attestify-crewai

CrewAI tool integration for Attestify — governed AI loop execution, receipts, audit trails, and Attestify Trust (no-wallet agent identity + signed evidence)

attestify-langchain

LangChain toolkit for Attestify — governed AI loop execution, receipts, audit trails, and Attestify Trust (no-wallet agent identity + signed evidence)

attestify-langgraph

LangGraph node integration for Attestify — governed AI loop execution, receipts, audit trails, and Attestify Trust (no-wallet agent identity + signed evidence)

claude-cost-tracker

Local-first cost observability for Claude Code sessions

kitaru-openai-agents

OpenAI Agents SDK recording adapter for Kitaru.

kitaru-pydantic-ai

PydanticAI recording and replay adapter for Kitaru.

legal-rag-audit

Endpoint-based evaluation harness for legal RAG systems

llamastat

nvtop for llama.cpp: an interactive terminal dashboard for llama-server inference monitoring

llm-agent-trace

Zero-dependency LLM call tracer — patch once, trace everything

llm-anthology

A local, private anthology of every AI coding session — browse, search, spawn-tree, faithfully export and safely manage ChatGPT / Claude / Gemini / Codex history. No network, no egress.

llmtivo

LLMTivo - record real LLM calls once, replay them forever, on the filesystem

memstrata-langchain-governance

LangChain audit callback integration for MemStrata Governance

ram-rag-lens

An open-source, pip-installable Python library that quantitatively evaluates RAG pipelines using LLM-as-a-Judge methodology.

vigilagent-sdk

Monitor, debug, and optimize your AI agents. Track LLM calls, costs, latency, and tool usage in real-time.

stepglass

See exactly what your AI agent did. Lightweight tracing + a local dashboard for debugging LangChain agent runs.

@thinkhive/sdk

ThinkHive SDK v4.0 - AI agent observability with business metrics, ROI analytics, and 25+ trace format support

@agentegrity/openai-agents

Zero-config agentegrity adapter for the OpenAI Agents JS SDK (@openai/agents). Mirrors the Python agentegrity.openai_agents module.

@oai-statsig/react-bindings

Statsig helps you move faster with feature gates (feature flags), and/or dynamic configs. It also allows you to run A/B/n tests to validate your new features and understand their impact on your KPIs. If you're new to Statsig, check out our product and cre

More in Agent frameworks & infrastructure