Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 52 of 57

n8n-nodes-llm-observability

n8n community nodes: Standalone AI Agent with multi-provider LLM support and integrated Langfuse/Phoenix/Helicone observability

@silverprotocol/openai-agents

AgJSON normalizer for the OpenAI Agents SDK (RunItemStreamEvent / Responses streaming → AgEvent[]) — fixture-tested.

@deepseek-ai/dsh-token-meter

Replay-aware token measurement service (ctx.tokenMeter) for the DeepSeek Harness

@deepseek-ai/dsh-session-stats

Whole-log conversation counts and wall times projection (sessionStats) for the DeepSeek Harness

@deepseek-ai/dsh-message-feedback

Lifecycle-bound per-message rating and note sidecar for the DeepSeek Harness

@deepseek-ai/dsh-client-ui-trajectory

Trajectory event ledger with an interactive timing overview: pure-consumer plugin registering into the conversation ViewMap (no service)

@deepseek-ai/dsh-session-telemetry-otel

OpenTelemetry backend for the DeepSeek Harness telemetry seam: hands captured session records to the OTel JS SDK's log pipeline

@deepseek-ai/dsh-session-telemetry

Telemetry seam for the DeepSeek Harness: session-event capture, projection, redaction, and handoff to a reporting backend

smthrs

Multi-agent workflows with full observability and time travel: watch every step live, rewind, fork, and replay any run. Claude Code, Codex, Gemini, Hermes, OpenClaw, any model or harness.

@nestjs-agentic/evaluation

Automated agent benchmarking, trajectory scoring, and LLM-as-a-Judge evaluation framework for nestjs-agentic.

@memora-hq/memora-core

Memora SDK: verifiable agent provenance via gateway (baseUrl + apiKey). Write, query, read, verify.

agent-ledger-adapters

One-liner audit trails + policy enforcement for LangChain, LlamaIndex and OpenAI Agents — zero dependencies

agent-trajectory-diff

A developer-first package to evaluate and find regressions in agent trajectories

agentabacus

Local-first analytics for AI coding agents. Reads the session logs already on your disk.

agentic-logger

Structured logging for Coding Agents - write once, read efficiently

agenticops-control-tower

Unified control plane and operations console for the DeepAgentLabs ecosystem.

agentlens-cli

Find out why your coding agent is expensive — offline waste detection, loop detection and skill generation across 7 coding agents.

agentplane-control-plane

The AgentPlane control plane: stores Chronicle traces and serves TokenOps governance over HTTP.

agentrelbench

k-run wrapper for the EnterpriseOps-Gym benchmark: drives evaluate.py k times per task, archiving per-run DB state exports plus a batch manifest.

agentrunproof

Provider-free Runner regression probes for the OpenAI Agents SDK

amigo-bedrock

Governed Bedrock invocation — pre-check, invoke, evidence, record, trace (AIGP-aligned)

chap-langgraph

CHAP bridge for LangGraph: emit hash-linked audit envelopes around human-in-the-loop interrupts.

chap-llama-index

CHAP adapter for LlamaIndex Workflows: record a human-decision envelope when a step pauses for human input.

coding-agent-cost

Estimate AI coding agent (Claude Code / Codex CLI) token usage and cost from local logs

crosstalk-agents

Stream-level supervision for AI agents: watch live token streams, interrupt mid-generation, gate tool actions pre-execution. Name reservation - the open-source release lands here in September 2026.

dsh-llm-cost

LLM API pricing registry and cost estimation, shared by the DeepSeek Harness dsh-llm-cost plugin and Python agent systems.

gosce-llm-orchestration-agent-opentelemetry-api

Run a prompt through a LangChain (system + human) chain over Gemini on Vertex AI; optional LangSmith tracing.

gpt-budget-guard

Budget checks for OpenAI API costs, locally and in GitHub Actions.

grafana-agento11y-hermes

Unofficial Grafana Agent Observability plugin for Hermes Agent. Records LLM calls and tool executions as generations and OTel traces+metrics.

kitaru-langgraph

LangGraph recording and replay adapter for Kitaru.

langgraph-checkpoint-cli

A terminal CLI for browsing LangGraph checkpoints — list threads, walk state history, inspect and diff checkpoints, no code required.

langgraph-scenario-lab

Behavioral scenario testing for production LangGraph applications

llm-metrics

Trace LLM calls with a decorator — non-blocking, fail-silent observability SDK.

llmapi-tracker

Local-first API usage and cost tracking platform

llmexer

Framework and CLI utility to create and curate datasets (e.g., publications, metadata, LLM prompts, etc.) and orchestrate (design, run, evaluate) various LLM experiments on them

llmlatency

Measured latency, uptime and outages of AI inference APIs (OpenAI, Anthropic, Gemini and 40+) from 4 regions — client for the open llmlatency.dev dataset

mistralai-evaluations

Mistral AI Studio evaluation tools

mistralai-workflows-plugins-evaluations

Evaluation plugin for Mistral Workflows - run evaluations as parallel Temporal activities

ollama-llmwatch

See what your local Ollama model is actually doing: live prefill progress, ETA, tok/s, and how long a whole agent turn takes.

ollama-usage

Log, budget, and visualise token usage for local Ollama models.

pytest-agent-trace

pytest for AI agents — trajectory-level recording, replay and assertions for LangGraph/LangChain agents

rag-governance-eval

Evaluation and quality-gating layer for RAG pipelines governed by Microsoft's Agent Governance Toolkit (agent-rag-governance)

ragassure

Identity-aware contract testing and release assurance for enterprise RAG systems

ragsanity

Zero-dependency RAG sanity check: score faithfulness, relevancy, and completeness in one function call. No API key, no ML libraries, no reference answers required.

render-langgraph

Zero-config LangGraph graph visualizer

@ai-agentree/sdk

AI Agentree TypeScript SDK — AI Agent Decision Tracing

inductio

A content-addressed, auditable foundation for LLM agent systems.

deepeval

The LLM Evaluation Framework for TypeScript

llm-interceptor

Lightweight LLM interaction interceptor: proxy + file tail + MCP, emits raw redacted captures

@rollyjoely/mlflow-langchain

LangChain integration package for MLflow Tracing

@lelemondev/sdk

Automatic LLM observability. Wrap your client, everything is traced.

@wix/pathgrade

Evaluate whether AI agents discover and use your skills correctly

@agentic-kit/pi-ext-run-log

pi extension that mirrors every session entry into an @agentic-kit/run-log store, verbatim and in order

agent-cost-tracker

Token, cache and context observability for AI coding agents.

agent-timetravel

Time-travel debugging for AI agents — OTel-in / replay-out.

agent-transcripts

Capture coding-agent transcripts to disk and expose them as SQL.

agent-ultra-kit

Agent-agnostic adversarial panel, ULTRA build loop, command broker, and proof gates for AI agent systems

agentcheck-ai

Behavioral testing for AI agents

gosce-traced-multi-llm-proxy

Proxy Gemini (Vertex AI) completions wrapped in OpenTelemetry trace spans; returns the answer plus the trace/span id.

loupe-agent

Open-source debugger for AI agents.

opentelemetry-instrumentation-genai-qwen-agent

OpenTelemetry Official Qwen-Agent instrumentation

opentelemetry-instrumentation-genai-smolagents

OpenTelemetry smolagents instrumentation

parlot-instrumentation-langgraph

Parlot instrumentation for LangGraph / LangChain — same turn model as voice AI agents

quanllm-harness

Plugin-driven agentic verification and recovery harness

soif-llm

Estimate the water footprint of LLM prompts, the way you estimate their cost.

sonar-tracing-langchain

LangChain / LangGraph adapter for sonar-tracing.

voice-agent-eval

Post-call evaluation for voice agents using audio and transcript signals.

warden-agent

Deterministic observability for AI agent trajectories and tool execution failures.

@libretto/openai

A drop-in replacement of the official `OpenAI` client for sending events to Libretto.

@dharma-ai-labs/agent-fabric

Local-first CLI for connecting coding agents to Dharma evaluation, orchestration, and signed Skill delivery

langgraph-inspector

Live terminal debugger and offline ASCII graph renderer for LangGraph.js — no LangSmith account, no network calls.

@cognipeer/observability

Ship agent traces from OpenAI Agents, Claude Agent SDK, LangChain, LangGraph, Vercel AI SDK, n8n and any OpenTelemetry-instrumented agent into Cognipeer Console.

agent-budget-semantics

Differential testing of budget enforcement semantics across AI agent frameworks

agent-failure-registry

Agent Failure Registry - a neutral namespace and crosswalk for AI agent failure modes

agent-reliability

Vendor-neutral reliability engineering for autonomous AI agents.

agent-visualizer

Watch your multi-agent AI workflow as a 2D retro pixel-art game

agentgrader

Provider-agnostic QA testing for LLM-based agents: accuracy, robustness, fairness, and security checks with pass/fail reporting.

deepeval-vllm

Compatibility wrapper for DeepEval's LocalModel that adds native structured output support.

gosce-ai-getvda-anthropic-lm-format-enforcer-mcp-5364ea

Proxy Gemini (Vertex AI) completions wrapped in OpenTelemetry trace spans; returns the answer plus the trace/span id.

langgraph-ledger

Full, tamper-evident traceability for LangGraph agents — a port of DeepSeek Harness (dsh)'s traceability design: hash-chained, content-addressed, replayable, rewindable.

llm-argus

Auto-instrumenting observability SDK for LLM inference.

llm-burnwatch

Local, zero-dependency cost tracking and anomaly detection for LLM/agent calls

meridian-storage-evidence

Meridian V1 evidence Catalog, audit, lineage, provenance, and OTel bridge contracts

model-info-amazon-bedrock

Python wrapper to simplify fetching information about models on Amazon Bedrock - particularly pricing

openai-radar

OpenAI infrastructure FinOps SDK — scan, analyze, and export your OpenAI org resources

ragarena

⚡ Evaluate & benchmark RAG strategies, LLMs and embedding models across every popular provider — 18 strategies (incl. graph + multimodal), 100+ models, built-in dashboard.

ragjudge

Evals for RAG pipelines — typed, async, honest.

@turns/dev

Upload local AI agent sessions to turns.dev

@apostl-dev/pulse-sdk

Server-side analytics for AI agent traffic

abslang

Agent Behavior Specification — describe, run, and evaluate AI agent interactions from the command line

@opentelemetry/otlp-exporter-base

OpenTelemetry OTLP Exporter base (for internal use only)

@deepseek-ai/dsh-llm-replay

Replay LLM plugin: short-circuits llm/stream with model chunks reconstructed from a recorded session JSONL (keyless snapshot tests)

@gosce/llm-observability-orchestration-agent-langchain

Run a prompt through a LangChain (system + human) chain over Gemini on Vertex AI; optional LangSmith tracing.

@gosce/llm-orchestration-agent

Run a prompt through a LangChain (system + human) chain over Gemini on Vertex AI; optional LangSmith tracing.

@graphmind-ai/langgraph

LangChain / LangGraph adapter for the GraphMind live agent debugger: a callback handler that streams the run tree as a live graph, plus tool wrappers that honour pause / step / inject / retry / abort gates.

agent-observatory

Local observability dashboard for Codex and Claude Code multi-agent workflows

dsh-plugin-agent-dashboard

Multi-agent dashboard skill for DeepSeek Harness -- status reports, task tracking, progress visualization

is-agentic

Score agent readiness with readable terminal reports and structured JSON.

@agentic-kit/metering

harness-neutral metering for an agent run — the Constructive gateway as a model endpoint, run identity headers, and usage reporting

@agentic-kit/run-log

The append-only agent run log — pi session entries stored verbatim under a run/seq wrapper, with projections for transcript, usage, tool state and resumable sessions

More in Agent frameworks & infrastructure