Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 47 of 57
Haystack integration for datadog
Counterfactual evaluation toolkit for RAG systems
Offline-first, framework-agnostic LLM evaluation primitives.
Future AGI is an all-in-one platform that helps you build reliable AI agents. It lets you simulate scenarios, catch errors, and keep your agents safe so they wo
LangChain adapter for emitting AeroGraph traces from agent workflows.
A testing library for agents
AgentInsight integration for LangChain
Cap every agent run before it starts: estimate cost, set a hard ceiling that stops the run, rescue stuck agents. Local, MIT, nothing uploaded.
Runtime signal adapters for AI Presence Kit.
A timeline visualizer for Railtracks agentic flows
LangGraph middleware for AgentTrace - automatic tracing of node executions
Unified observability dashboard for the agent stack — metrics, audit, escalations, policy, health
Deterministic record/replay of LLM and tool calls for AI agents, captured into human-readable cassettes.
一个用于测试的 Mock LLM 服务器,支持 OpenAI、Anthropic 和 Responses API 格式
Drop-in LLMOps observability. Self-hosted. SQLite-default. Persona-aware. Multi-judge consensus.
A magnifying glass for your AI agent — TypeScript SDK.
ZokLens JS/TS SDK — LLM & Agent Observability via OpenTelemetry
Logging and tracing layer for AgentsKit agents.
Raindrop integration for Vercel's Eve agent framework (OTel traces to Raindrop + Workshop)
Visualize and follow your agentic-surfaces workflows in the browser — run `npx agentic-surfaces` from your repo to open the live dashboard.
Agentic Time: measure an AI agent's internal time and 7-state health from per-channel state deltas, in the browser, on the edge, or in Node — via a tiny WASM build of the emergent-time Rust crate.
Live observability and visualization for AI agent activity.
A State-Aware Rollback and Trajectory-Replanning Framework for Autonomous LLM Agents
Zespan observability middleware for CrewAI
Nebuly-AI server side SDK for JavaScript
LangChain.js integration for Prefactor observability
Drop-in audit/compliance callback handlers for LangChain, LangGraph, CrewAI, AutoGen, Mastra, PydanticAI, and Microsoft Agent Framework. Streams events into the Ujex hash-chained audit ledger (IETF draft-sharif-agent-audit-trail-00). Resolves LangChain RF
Papaya AI agent trace capture SDK for production apps.
Evidence-grounded evaluator for AI agent trajectories — judge by verifying claims against real tool outputs, not LLM-judge vibes.
Real-time observability and auto-debugging toolkit for AI agents
Semantic diff engine for AI agent behavior — find the exact decision fork between two runs
AgentMetrics - self-hosted AI agent observability server
Track LLM costs across your AI agents
AI-powered observability SDK for Python applications
Live terminal TUI tracking Claude Code token usage and cost estimates
AgentMetrics observability integration for LangChain.js
AgentMetrics observability integration for Claude Managed Agents
AgentMetrics observability integration for AutoGen agents
AgentMetrics observability integration for CrewAI agents
AgentMetrics observability plugin for Hermes agents
AgentMetrics observability integration for LangChain agents
AgentMetrics observability integration for LlamaIndex agents
AgentMetrics observability integration for OpenAI Agents SDK
Shared event schema, pricing, and redaction for all agentmetrics integrations
Typed, auto-generated client for the OpenSwarm product-analytics ingest API.
A small package to fuzz test LLMs
Renamed to cdv — a research preview of a dual-loop evaluation architecture for agentic LLM systems.
Lightweight evaluation metrics for RAG (Hit@k, Recall@k, Precision@k, MRR, nDCG, and more)
AI Agent collaboration analytics - become your AI agent's boss, not its babysitter
Langfuse-style trace viewer for pi-agent — local-first, zero-setup, single-file HTML
The debugger for AI agents — add 2 lines, replay every run, get the plain-English root cause and exact fix in 30 seconds.
Deterministic release gates for AI agent runs, backed by synthetic fixtures.
Decision records for AI agents — log every fork your agent considered, rejected, and executed as replayable JSONL. Adapters for OpenAI Agents and LangGraph, an MCP host, a proxy layer, policy packs, and signed export packs.
Deterministic log templating on top of Drain3, packaged as an artifact for AI agents.
Shared sensing contract for AI-agent graders — the verdict/report/intent/handoff models and the Sense protocol behind AgentVision (eyes), Audel (ears) and Verel (brain).
Serverless radar for Amazon Bedrock — deploy a Lambda that logs prompt/latency to S3, then analyze it.
Run a prompt through a LangChain (system + human) chain over Gemini on Vertex AI; optional LangSmith tracing.
Zespan observability middleware for Microsoft AutoGen
Structured logger and CLI visualizer for AI agent traces (LLM calls, tool calls, reasoning)
CLI for tracking AI agent token usage
Explainable backend flows — automatic causal traces, decision evidence, and MCP tool generation for AI agents
Local agent status reference implementation
Production reliability suite for LangChain/LangGraph multi-agent systems
Lightweight SDK for tracking AI agent costs and reliability with AgentAxon
Cut AI agent token costs. One line of code. Drop-in supervisor for LangChain, LangGraph, AutoGen, CrewAI, and Smolagents.
Chrome DevTools for AI agents — capture, view, and debug agent traces
ForgeSight LangGraph adapter — auto-instrument LangGraph/LangChain agents via callbacks.
Haystack integration for opentelemetry
A lightweight, resilient, and production-ready evaluation engine to audit Retrieval-Augmented Generation (RAG) pipelines.
The **OpenTelemetry integration** for the [AI SDK](https://ai-sdk.dev/docs) collects tracing spans for AI SDK calls and sends them through your configured OpenTelemetry setup.
A leaderboard for your agentic-coding token usage — race your friends, not strangers.
Model-independent agentic benchmark harness for WrongStack (bundled smoke suite, Aider polyglot, SWE-bench Verified) with deterministic graders, harness fingerprinting, and run-to-run comparison reports.
Track AI agent costs, tokens, runtime and spending. Prevent runaway OpenAI, Anthropic, LangGraph and OpenRouter agents from exceeding budget.
A reliability harness for scheduled/autonomous agent systems: persist state across runs, prove liveness not just freshness, fail loud, and scan for the failure class before it ships.
Self-hosted AgentSave dashboard backend
A lightweight metrics exporter for LiteLLM — scrapes Prometheus metrics and serves JSON for dashboards like Homepage and Home Assistant.
Framework-agnostic record/replay test harness for LLM agents
Local-first, multi-provider tool that captures LLM API spend and exposes it to coding agents via MCP
Privacy-first, SQLite-first observability for local Ollama agents.
AgentLens: local-first diagnostics & observability for AI agents. Answers 'Why did my agent behave this way?' with one line and a single SQLite file.
A utility library to evaluate LLMs.
SigRank local telemetry agent — scans AI-platform logs, computes the 11-metric stack, publishes signed snapshots.
CLS observability SDK for LangChain/LangGraph — 6-layer span hierarchy, direct upload to Tencent Cloud CLS
WasmAgent Compliance Engine — TaskSpec-driven verification, local repair, and evidence export for LLM agent runs.
Fetch an OpenTelemetry trace from Axiom or Jaeger and render it into a single, deterministic Markdown file — structured for human reading and LLM agent consumption.
Provider-agnostic token profiler for LLM agent loops. See exactly where your context tokens go — and how many you waste re-sending the same thing.
Opt-in Langfuse trace sink for the Agent Architecture Board. The workerd-safe OTLP/HTTP exporter pointed at Langfuse's OpenTelemetry endpoint, so LLM generations and container tool spans stream unchanged on both the Cloudflare Worker and Node facades.
Agent evaluation and benchmarking for AgentsKit.
LangChain.js / LangGraph callback adapter for AgentInspect local execution trees and Evidence workflows
LangChain TS callback handler for SpendGuard budget guardrails
LangSmith tracing module and decorators for NestJS.
Enprompta framework instrumentation for LangChain — emits typed, nested OpenTelemetry spans that Enprompta ingests as first-class Retrieval/Tool/Agent spans.
Stratos AI agent telemetry (OpenLLMetry), Vercel AI auto-spans, LangChain inspect/fix, MCP proxy.
Vercel AI SDK telemetry integration for AgentInspect local traces and trajectory tests
OpenAI Agents JS processor integration for AgentInspect local traces and TraceContract checks
Agent observability and knowledge capture layer for AI coding tools.
Langfuse integration for Vercel AI SDK v7
Runtime loop and deadlock detection for multi-agent AI systems
Benchmark and observability harness for comparing agentic coding tools.
Reusable proof profile, report, graph, and witness-planning primitives.