Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 52 of 57
n8n community nodes: Standalone AI Agent with multi-provider LLM support and integrated Langfuse/Phoenix/Helicone observability
AgJSON normalizer for the OpenAI Agents SDK (RunItemStreamEvent / Responses streaming → AgEvent[]) — fixture-tested.
Replay-aware token measurement service (ctx.tokenMeter) for the DeepSeek Harness
Whole-log conversation counts and wall times projection (sessionStats) for the DeepSeek Harness
Lifecycle-bound per-message rating and note sidecar for the DeepSeek Harness
Trajectory event ledger with an interactive timing overview: pure-consumer plugin registering into the conversation ViewMap (no service)
OpenTelemetry backend for the DeepSeek Harness telemetry seam: hands captured session records to the OTel JS SDK's log pipeline
Telemetry seam for the DeepSeek Harness: session-event capture, projection, redaction, and handoff to a reporting backend
Multi-agent workflows with full observability and time travel: watch every step live, rewind, fork, and replay any run. Claude Code, Codex, Gemini, Hermes, OpenClaw, any model or harness.
Automated agent benchmarking, trajectory scoring, and LLM-as-a-Judge evaluation framework for nestjs-agentic.
Memora SDK: verifiable agent provenance via gateway (baseUrl + apiKey). Write, query, read, verify.
One-liner audit trails + policy enforcement for LangChain, LlamaIndex and OpenAI Agents — zero dependencies
A developer-first package to evaluate and find regressions in agent trajectories
Local-first analytics for AI coding agents. Reads the session logs already on your disk.
Structured logging for Coding Agents - write once, read efficiently
Unified control plane and operations console for the DeepAgentLabs ecosystem.
Find out why your coding agent is expensive — offline waste detection, loop detection and skill generation across 7 coding agents.
The AgentPlane control plane: stores Chronicle traces and serves TokenOps governance over HTTP.
k-run wrapper for the EnterpriseOps-Gym benchmark: drives evaluate.py k times per task, archiving per-run DB state exports plus a batch manifest.
Provider-free Runner regression probes for the OpenAI Agents SDK
Governed Bedrock invocation — pre-check, invoke, evidence, record, trace (AIGP-aligned)
CHAP bridge for LangGraph: emit hash-linked audit envelopes around human-in-the-loop interrupts.
CHAP adapter for LlamaIndex Workflows: record a human-decision envelope when a step pauses for human input.
Estimate AI coding agent (Claude Code / Codex CLI) token usage and cost from local logs
Stream-level supervision for AI agents: watch live token streams, interrupt mid-generation, gate tool actions pre-execution. Name reservation - the open-source release lands here in September 2026.
LLM API pricing registry and cost estimation, shared by the DeepSeek Harness dsh-llm-cost plugin and Python agent systems.
Run a prompt through a LangChain (system + human) chain over Gemini on Vertex AI; optional LangSmith tracing.
Budget checks for OpenAI API costs, locally and in GitHub Actions.
Unofficial Grafana Agent Observability plugin for Hermes Agent. Records LLM calls and tool executions as generations and OTel traces+metrics.
LangGraph recording and replay adapter for Kitaru.
A terminal CLI for browsing LangGraph checkpoints — list threads, walk state history, inspect and diff checkpoints, no code required.
Behavioral scenario testing for production LangGraph applications
Trace LLM calls with a decorator — non-blocking, fail-silent observability SDK.
Local-first API usage and cost tracking platform
Framework and CLI utility to create and curate datasets (e.g., publications, metadata, LLM prompts, etc.) and orchestrate (design, run, evaluate) various LLM experiments on them
Measured latency, uptime and outages of AI inference APIs (OpenAI, Anthropic, Gemini and 40+) from 4 regions — client for the open llmlatency.dev dataset
Mistral AI Studio evaluation tools
Evaluation plugin for Mistral Workflows - run evaluations as parallel Temporal activities
See what your local Ollama model is actually doing: live prefill progress, ETA, tok/s, and how long a whole agent turn takes.
Log, budget, and visualise token usage for local Ollama models.
pytest for AI agents — trajectory-level recording, replay and assertions for LangGraph/LangChain agents
Evaluation and quality-gating layer for RAG pipelines governed by Microsoft's Agent Governance Toolkit (agent-rag-governance)
Identity-aware contract testing and release assurance for enterprise RAG systems
Zero-dependency RAG sanity check: score faithfulness, relevancy, and completeness in one function call. No API key, no ML libraries, no reference answers required.
Zero-config LangGraph graph visualizer
AI Agentree TypeScript SDK — AI Agent Decision Tracing
A content-addressed, auditable foundation for LLM agent systems.
The LLM Evaluation Framework for TypeScript
Lightweight LLM interaction interceptor: proxy + file tail + MCP, emits raw redacted captures
LangChain integration package for MLflow Tracing
Automatic LLM observability. Wrap your client, everything is traced.
Evaluate whether AI agents discover and use your skills correctly
pi extension that mirrors every session entry into an @agentic-kit/run-log store, verbatim and in order
Token, cache and context observability for AI coding agents.
Time-travel debugging for AI agents — OTel-in / replay-out.
Capture coding-agent transcripts to disk and expose them as SQL.
Agent-agnostic adversarial panel, ULTRA build loop, command broker, and proof gates for AI agent systems
Behavioral testing for AI agents
Proxy Gemini (Vertex AI) completions wrapped in OpenTelemetry trace spans; returns the answer plus the trace/span id.
Open-source debugger for AI agents.
OpenTelemetry Official Qwen-Agent instrumentation
OpenTelemetry smolagents instrumentation
Parlot instrumentation for LangGraph / LangChain — same turn model as voice AI agents
Plugin-driven agentic verification and recovery harness
Estimate the water footprint of LLM prompts, the way you estimate their cost.
LangChain / LangGraph adapter for sonar-tracing.
Post-call evaluation for voice agents using audio and transcript signals.
Deterministic observability for AI agent trajectories and tool execution failures.
A drop-in replacement of the official `OpenAI` client for sending events to Libretto.
Local-first CLI for connecting coding agents to Dharma evaluation, orchestration, and signed Skill delivery
Live terminal debugger and offline ASCII graph renderer for LangGraph.js — no LangSmith account, no network calls.
Ship agent traces from OpenAI Agents, Claude Agent SDK, LangChain, LangGraph, Vercel AI SDK, n8n and any OpenTelemetry-instrumented agent into Cognipeer Console.
Differential testing of budget enforcement semantics across AI agent frameworks
Agent Failure Registry - a neutral namespace and crosswalk for AI agent failure modes
Vendor-neutral reliability engineering for autonomous AI agents.
Watch your multi-agent AI workflow as a 2D retro pixel-art game
Provider-agnostic QA testing for LLM-based agents: accuracy, robustness, fairness, and security checks with pass/fail reporting.
Compatibility wrapper for DeepEval's LocalModel that adds native structured output support.
Proxy Gemini (Vertex AI) completions wrapped in OpenTelemetry trace spans; returns the answer plus the trace/span id.
Full, tamper-evident traceability for LangGraph agents — a port of DeepSeek Harness (dsh)'s traceability design: hash-chained, content-addressed, replayable, rewindable.
Auto-instrumenting observability SDK for LLM inference.
Local, zero-dependency cost tracking and anomaly detection for LLM/agent calls
Meridian V1 evidence Catalog, audit, lineage, provenance, and OTel bridge contracts
Python wrapper to simplify fetching information about models on Amazon Bedrock - particularly pricing
OpenAI infrastructure FinOps SDK — scan, analyze, and export your OpenAI org resources
⚡ Evaluate & benchmark RAG strategies, LLMs and embedding models across every popular provider — 18 strategies (incl. graph + multimodal), 100+ models, built-in dashboard.
Evals for RAG pipelines — typed, async, honest.
Upload local AI agent sessions to turns.dev
Server-side analytics for AI agent traffic
Agent Behavior Specification — describe, run, and evaluate AI agent interactions from the command line
OpenTelemetry OTLP Exporter base (for internal use only)
Replay LLM plugin: short-circuits llm/stream with model chunks reconstructed from a recorded session JSONL (keyless snapshot tests)
Run a prompt through a LangChain (system + human) chain over Gemini on Vertex AI; optional LangSmith tracing.
Run a prompt through a LangChain (system + human) chain over Gemini on Vertex AI; optional LangSmith tracing.
LangChain / LangGraph adapter for the GraphMind live agent debugger: a callback handler that streams the run tree as a live graph, plus tool wrappers that honour pause / step / inject / retry / abort gates.
Local observability dashboard for Codex and Claude Code multi-agent workflows
Multi-agent dashboard skill for DeepSeek Harness -- status reports, task tracking, progress visualization
Score agent readiness with readable terminal reports and structured JSON.
harness-neutral metering for an agent run — the Constructive gateway as a model endpoint, run identity headers, and usage reporting
The append-only agent run log — pi session entries stored verbatim under a run/seq wrapper, with projections for transcript, usage, tool state and resumable sessions