Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 47 of 57

datadog-haystack

Haystack integration for datadog

evalkit-rag

Counterfactual evaluation toolkit for RAG systems

llmcalibre

Offline-first, framework-agnostic LLM evaluation primitives.

Future AGI

Future AGI is an all-in-one platform that helps you build reliable AI agents. It lets you simulate scenarios, catch errors, and keep your agents safe so they wo

@aerograph/adapter-langchain

LangChain adapter for emitting AeroGraph traces from agent workflows.

@agest/core

A testing library for agents

@agentinsight-sdk/langchain

AgentInsight integration for LangChain

runcap

Cap every agent run before it starts: estimate cost, set a hard ceiling that stops the run, rescue stuck agents. Local, MIT, nothing uploaded.

@ai-presence/adapters

Runtime signal adapters for AI Presence Kit.

@railtownai/railtracks-timeline

A timeline visualizer for Railtracks agentic flows

@agenttrace-io/middleware-langgraph

LangGraph middleware for AgentTrace - automatic tracing of node executions

agentobserve-py

Unified observability dashboard for the agent stack — metrics, audit, escalations, policy, health

agenttape

Deterministic record/replay of LLM and tool calls for AI agents, captured into human-readable cassettes.

llm-mock-api

一个用于测试的 Mock LLM 服务器,支持 OpenAI、Anthropic 和 Responses API 格式

omnismart-rageval

Drop-in LLMOps observability. Self-hosted. SQLite-default. Persona-aware. Multi-judge consensus.

loupe-ai

A magnifying glass for your AI agent — TypeScript SDK.

zoklens

ZokLens JS/TS SDK — LLM & Agent Observability via OpenTelemetry

@agentskit/observability

Logging and tracing layer for AgentsKit agents.

@raindrop-ai/eve

Raindrop integration for Vercel's Eve agent framework (OTel traces to Raindrop + Workshop)

agentic-surfaces

Visualize and follow your agentic-surfaces workflows in the browser — run `npx agentic-surfaces` from your repo to open the live dashboard.

@ruvector/emergent-time

Agentic Time: measure an AI agent's internal time and 7-state health from per-channel state deltas, in the browser, on the edge, or in Node — via a tiny WASM build of the emergent-time Rust crate.

agentperiscope

Live observability and visualization for AI agent activity.

agents-driftguard

A State-Aware Rollback and Trajectory-Replanning Framework for Autonomous LLM Agents

zespan-crewai

Zespan observability middleware for CrewAI

@nebuly-ai/nebuly-js-sdk

Nebuly-AI server side SDK for JavaScript

@prefactor/langchain

LangChain.js integration for Prefactor observability

@ujexdev/compliance-callback

Drop-in audit/compliance callback handlers for LangChain, LangGraph, CrewAI, AutoGen, Mastra, PydanticAI, and Microsoft Agent Framework. Streams events into the Ujex hash-chained audit ledger (IETF draft-sharif-agent-audit-trail-00). Resolves LangChain RF

@papaya-ai/tracing

Papaya AI agent trace capture SDK for production apps.

agent-attest

Evidence-grounded evaluator for AI agent trajectories — judge by verifying claims against real tool outputs, not LLM-judge vibes.

agent-observer

Real-time observability and auto-debugging toolkit for AI agents

agentdelta

Semantic diff engine for AI agent behavior — find the exact decision fork between two runs

agentmetrics-server

AgentMetrics - self-hosted AI agent observability server

agentspend-tracker

Track LLM costs across your AI agents

aiops-agent

AI-powered observability SDK for Python applications

stradiz-claude-tokens

Live terminal TUI tracking Claude Code token usage and cost estimates

agentmetrics-langchain

AgentMetrics observability integration for LangChain.js

agentmetrics-anthropic

AgentMetrics observability integration for Claude Managed Agents

agentmetrics-autogen

AgentMetrics observability integration for AutoGen agents

agentmetrics-crewai

AgentMetrics observability integration for CrewAI agents

agentmetrics-hermes

AgentMetrics observability plugin for Hermes agents

agentmetrics-langchain

AgentMetrics observability integration for LangChain agents

agentmetrics-llamaindex

AgentMetrics observability integration for LlamaIndex agents

agentmetrics-openai-agents

AgentMetrics observability integration for OpenAI Agents SDK

agentmetrics-shared

Shared event schema, pricing, and redaction for all agentmetrics integrations

swarm-analytics

Typed, auto-generated client for the OpenSwarm product-analytics ingest API.

llm-fuzzer

A small package to fuzz test LLMs

loopllm

Renamed to cdv — a research preview of a dual-loop evaluation architecture for agentic LLM systems.

rag-eval-lite

Lightweight evaluation metrics for RAG (Hit@k, Recall@k, Precision@k, MRR, nDCG, and more)

agentboss

AI Agent collaboration analytics - become your AI agent's boss, not its babysitter

pi-trace-extension

Langfuse-style trace viewer for pi-agent — local-first, zero-setup, single-file HTML

vorlo-trace

The debugger for AI agents — add 2 lines, replay every run, get the plain-English root cause and exact fix in 30 seconds.

agent-proof-kit

Deterministic release gates for AI agent runs, backed by synthetic fixtures.

deliberate-sdk

Decision records for AI agents — log every fork your agent considered, rejected, and executed as replayable JSONL. Adapters for OpenAI Agents and LangGraph, an MCP host, a proxy layer, policy packs, and signed export packs.

agentic-log-analyser

Deterministic log templating on top of Drain3, packaged as an artifact for AI agents.

agentsensory

Shared sensing contract for AI-agent graders — the verdict/report/intent/handoff models and the Sense protocol behind AgentVision (eyes), Audel (ears) and Verel (brain).

aws-bedrock-radar

Serverless radar for Amazon Bedrock — deploy a Lambda that logs prompt/latency to S3, then analyze it.

gosce-llm-observability-orchestration

Run a prompt through a LangChain (system + human) chain over Gemini on Vertex AI; optional LangSmith tracing.

zespan-autogen

Zespan observability middleware for Microsoft AutoGen

ai-agent-trace

Structured logger and CLI visualizer for AI agent traces (LLM calls, tool calls, reasoning)

@lmoesle/token-usage-cli

CLI for tracking AI agent token usage

footprintjs

Explainable backend flows — automatic causal traces, decision evidence, and MCP tool generation for AI agents

agent-status

Local agent status reference implementation

agentarmour-toolkit

Production reliability suite for LangChain/LangGraph multi-agent systems

agentaxon

Lightweight SDK for tracking AI agent costs and reliability with AgentAxon

agentsave

Cut AI agent token costs. One line of code. Drop-in supervisor for LangChain, LangGraph, AutoGen, CrewAI, and Smolagents.

agentx-trace

Chrome DevTools for AI agents — capture, view, and debug agent traces

forgesight-adapters-langgraph

ForgeSight LangGraph adapter — auto-instrument LangGraph/LangChain agents via callbacks.

opentelemetry-haystack

Haystack integration for opentelemetry

rag-scorecard

A lightweight, resilient, and production-ready evaluation engine to audit Retrieval-Augmented Generation (RAG) pipelines.

@ai-sdk/otel

The **OpenTelemetry integration** for the [AI SDK](https://ai-sdk.dev/docs) collects tracing spans for AI SDK calls and sends them through your configured OpenTelemetry setup.

@tokenboard/cli

A leaderboard for your agentic-coding token usage — race your friends, not strangers.

@wrongstack/bench

Model-independent agentic benchmark harness for WrongStack (bundled smoke suite, Aider polyglot, SWE-bench Verified) with deterministic graders, harness fingerprinting, and run-to-run comparison reports.

@painitehq/ai-budget

Track AI agent costs, tokens, runtime and spending. Prevent runaway OpenAI, Anthropic, LangGraph and OpenRouter agents from exceeding budget.

agentliveness

A reliability harness for scheduled/autonomous agent systems: persist state across runs, prove liveness not just freshness, fail loud, and scan for the failure class before it ships.

agentsave-dashboard

Self-hosted AgentSave dashboard backend

litellm-pulse

A lightweight metrics exporter for LiteLLM — scrapes Prometheus metrics and serves JSON for dashboards like Homepage and Home Assistant.

llm-tape

Framework-agnostic record/replay test harness for LLM agents

llm-usage-mcp

Local-first, multi-provider tool that captures LLM API spend and exposes it to coding agents via MCP

ollama-telemetry

Privacy-first, SQLite-first observability for local Ollama agents.

pyagentlens

AgentLens: local-first diagnostics & observability for AI agents. Answers 'Why did my agent behave this way?' with one line and a single SQLite file.

r-llm-evals

A utility library to evaluate LLMs.

sigrank-agent

SigRank local telemetry agent — scans AI-platform logs, computes the 11-metric stack, publishes signed snapshots.

tencentcloud-cls-sdk-langchain

CLS observability SDK for LangChain/LangGraph — 6-layer span hierarchy, direct upload to Tencent Cloud CLS

@wasmagent/compliance

WasmAgent Compliance Engine — TaskSpec-driven verification, local repair, and evidence export for LLM agent runs.

@useairfoil/traceview

Fetch an OpenTelemetry trace from Axiom or Jaeger and render it into a single, deterministic Markdown file — structured for human reading and LLM agent consumption.

@metacapsin/tokenlens

Provider-agnostic token profiler for LLM agent loops. See exactly where your context tokens go — and how many you waste re-sending the same thing.

@cat-factory/observability-langfuse

Opt-in Langfuse trace sink for the Agent Architecture Board. The workerd-safe OTLP/HTTP exporter pointed at Langfuse's OpenTelemetry endpoint, so LLM generations and container tool spans stream unchanged on both the Cloudflare Worker and Node facades.

@agentskit/eval

Agent evaluation and benchmarking for AgentsKit.

@agent-inspect/langchain

LangChain.js / LangGraph callback adapter for AgentInspect local execution trees and Evidence workflows

@spendguard/langchain

LangChain TS callback handler for SpendGuard budget guardrails

@nest-langchain/langsmith

LangSmith tracing module and decorators for NestJS.

@enprompta/instrument-langchain

Enprompta framework instrumentation for LangChain — emits typed, nested OpenTelemetry spans that Enprompta ingests as first-class Retrieval/Tool/Agent spans.

@intelli-1113/stratos-sdk

Stratos AI agent telemetry (OpenLLMetry), Vercel AI auto-spans, LangChain inspect/fix, MCP proxy.

@agent-inspect/ai-sdk

Vercel AI SDK telemetry integration for AgentInspect local traces and trajectory tests

@agent-inspect/openai-agents

OpenAI Agents JS processor integration for AgentInspect local traces and TraceContract checks

@tonyclaw/agent-inspector

Agent observability and knowledge capture layer for AI coding tools.

@langfuse/vercel-ai-sdk

Langfuse integration for Vercel AI SDK v7

agent-watchdog

Runtime loop and deadlock detection for multi-agent AI systems

bench-my-harness

Benchmark and observability harness for comparing agentic coding tools.

agentic-proofkit

Reusable proof profile, report, graph, and witness-planning primitives.

More in Agent frameworks & infrastructure