Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 53 of 57

@moldea.ai/adapter-langgraph

Deterministic runtime evidence and diagnostics for LangGraph TypeScript workflows.

SafeLLM4SE

Statistical evaluation and reporting for LLM-based software engineering systems.

agent-audit-record

Reference emitter for the agent-audit OpenTelemetry semantic convention: proposed -> decided -> executed.

agent-blame

LLM agent execution tracer with failure attribution

agent-session-otel

Discover local Claude Code / Codex session logs and export them as OpenTelemetry GenAI-compatible traces.

agentbase-agentops-langchain

Automatic LangChain Agent observability for AgentBase AgentOps

agentcore-metering

Unified observability for LLM/text, image, and video calls (Django adapter)

agenteval-python

Reliability and audit-ready testing for LLM agents (Python bridge to the AgentEval engine).

agentic-otel

Normalization layer for OpenTelemetry GenAI agent traces. Single parsing pass, framework detection, span classification with confidence scores.

agentjury

Peer review for AI agents using independent model juries and deterministic aggregation.

agentnorm

Behavioural monitoring for AI agents: runtime baselines, scope enforcement and cold-start-safe anomaly detection.

agents-traces

Append-only JSONL session traces, assemble, and MCP inspect tools

ark-agent-runtime

ARK Runtime SDK — attach ARK decision telemetry, cost attribution, and (experimental) constrained supervision around your existing agent.

causa-rag-client

Thin HTTP client for the RAG debugging/eval platform — register an external RAG, upload datasets, run experiments, fetch results.

causal-agent-verifier

A first-principles evaluation framework for LLM agents based on causal invariants, state transitions, and counterfactuals.

claude-session-diagnostics

Cost and prompt-cache diagnostics for Claude Code session transcripts

codex-agent-hour-tracker

Measure cumulative Codex and Claude Code agent runtime without decoding conversation content.

digline-anthropic

Anthropic target and judges for digline: prompt file in, priced Response out.

digline-bedrock

Amazon Bedrock target and judges for digline, on the Converse API.

digline-openai

OpenAI target and judges for digline, at any OpenAI-compatible endpoint.

distributed-llm-observatory

Open-source framework for building agent configurations, testing AI agents, and comparing reproducible AI-system observations.

insightfactory-databricks-langgraph-tracer

LangGraph tracing for Databricks MLflow

llm-panel

Put one question to several LLMs independently and read every answer in full -- parallel judges, failure classes, cost, and an anonymised rebuttal round

llm-waterpipe

Evaluation pipeline for LLM watermark detection experiments

llmgauge

Practical local LLM evaluation on real hardware

llmtrack-sdk

Server-side SDK for tracking LLM token usage, costs, and observability data in LLMtrack

microsoft-agents-testing

Library for building agent tests using Microsoft Agents SDK

placeboagent

Fake AG-UI agent backend for testing. Deterministic scripted event streams, session record & replay, failure injection. No LLM needed.

pytest-agentreplay

Regression tests for AI agents. Record once, replay offline, catch behavioural regressions.

pythonllmhook

Runtime-scoped LLM hooks for verified Python handlers and source patches

rag-eval-toolkit

RAG evaluation toolkit with readable LLM-judge prompts and local-first (Ollama) support.

ragdiag

RAG evaluation and root-cause diagnosis SDK/CLI

ragval

The complete RAG evaluation library. Scores, diagnoses, and domain-aware.

rhesis-haystack

Haystack integration for rhesis

risenlab-agentlogs

Schema definitions and utilities for the AgentLogs dataset

sannex-agent

Official Python Telemetry & Monitoring Client for Sannex AI Operations

tencentcloud-agentobs-sdk-agentscope

Tencent Cloud CLS observability for AgentScope agents

tencentcloud-agentobs-sdk-openai-agent

CLS observability SDK for OpenAI Agents SDK — per-turn trace model, direct upload to Tencent Cloud CLS

tls-llm-observer

OpenTelemetry-first LLM tracing SDK with OpenAI instrumentation

trulens-apps-claude

Claude Code hook instrumentation plugin for TruLens.

trulens-providers-ollama

Library to systematically track and evaluate LLM based applications.

@datadog/libdatadog

WASM Node.js bindings for libdatadog

agent2perfetto

Convert agent session logs (Claude Code JSONL) into Perfetto/Chrome trace JSON for ui.perfetto.dev

agentic-dataset-conformance

AD-001..AD-015: the agentic-dataset governance contract as language-neutral executable conformance vectors, plus a runner that needs no access to an implementation's internals

agentlens-evaluator

Framework-agnostic observability, debugging, and deterministic issue detection for AI agents.

agno-metrics-rag

Ragas-inspired RAG quality evaluation, synthetic dataset generation, and continuous runtime observability for Agno.

cdzzy-agenttest

The testing framework for AI agents — think pytest, but for agents.

llm-regress-ci

Regression testing for LLM applications (CI for prompts)

testimony-langgraph

Emit a Testimony Record from a LangGraph human-in-the-loop flow, so the record can say who approved.

agentcassette

Deterministic semantic record and replay for LLM agent tests

@sanning/anchor-langchain

Anchor your LangChain.js agent's run tree as it executes: one callback handler turns every chain, LLM and tool step into a signed evidence envelope, sent to Sanning and witnessed on an interval to a permanent public record. Prompts and outputs are hashed

@graphmind-ai/openai

OpenAI Node SDK adapter for the GraphMind live agent debugger: wrap your OpenAI client and tools to stream execution events and honour pause / step / inject / retry / abort gates.

@chatu-ai/builder-sdk-mock

Mock BuilderClient for @chatu-ai/builder-sdk: scripted event replay with latency & disconnect injection

mlflow-vercel

Vercel AI SDK integration package for MLflow Tracing

@singleaxis/fabric

Customer-controlled OpenTelemetry capture SDK for reconstructing AI agent activity.

agent-safety-bench

Multi-step agent safety compliance benchmarking — finds the critical depth D* where LLM agents fail to maintain safety policies across sequential interactions.

agentbelt-budget

A lightweight, in-process circuit breaker for LLM token and cost budgets. Part of the agentbelt suite.

agentbelt-loop

Detect repeat, cyclic, and stagnation loops in AI agent tool-call sequences, with cross-session pattern learning. Part of the agentbelt suite.

agentbelt-sampler

Agent-aware span sampling: keeps errors, cost/latency outliers, retries, and novel calls; samples the rest. Part of the agentbelt suite.

agentbelt-suite

Budget circuit breaking, loop detection, and span sampling for AI agent workflows, in one install.

agentlens-audit

Multi-agent audit CLI: cost governance, decision audit, evidence chain, shadow agent detection, compliance

agentship-observability

AgentShip observability — the OpenTelemetry implementation of the Observer port: span-tree exporter pipeline (Phoenix/Langfuse/LangSmith), the one-per-process LiteLLM cost/token callback, and the studio launcher. The vendor-free contract lives in agentship-core.

anlyon-langchain

LangChain integration for Anlyon: actions as tools, and chains and agents traced as Anlyon runs

anlyon-llamaindex

LlamaIndex integration for Anlyon: actions as tools, plus a vector store on the memory namespace

cls-agent-observability

OpenAI Agents SDK 的腾讯云 CLS Agent 可观测接入 —— 一行初始化,无需手动埋点

langgraphics-coze

Debug LangGraph-based agents visually - live execution graph, node inspection, and run replay, all from a single line.

llm-agent-metering

Language-agnostic LLM cost metering via HTTP proxy — track spend per customer and feature

llm-forensic-timeline

Standardized methodology and dataset for evaluating LLM-based digital forensic timeline analysis

llm-output-repair

FSM syntactic engine to repair truncated/broken JSON from LLMs (streaming, Pydantic, zero-regex core)

llm-reliability

Evidence-based reliability analyzer and root-cause diagnosis engine for LLM, RAG, and Agent systems.

probatio-llm

A pytest plugin for regression-testing LLM applications: metamorphic relations and flakiness statistics.

proveai-agentpulse

Observability and drift detection for multi-agent systems (AutoGen, LangChain/LangGraph)

pydantic-ai-governor

Agent Execution Evidence for Pydantic AI, by Sentience Governor

ragscope-sdk

An open source observability RAG framework.

tencentcloud-agentobs-sdk-langchain

CLS observability SDK for LangChain/LangGraph — 6-layer span hierarchy, direct upload to Tencent Cloud CLS

watchdog-agent-sdk

Fail-open reliability monitoring for unattended agent jobs

agentworth

Discover, normalize, and understand AI-agent histories locally.

@alvin0/ai-agent-sdk-observability-otel

Universal OpenTelemetry API bridge for correlated ai-agent-sdk spans, metrics, and logs

hannah-agent-runtime

A cross-agent-runtime unified event and policy layer for observing, constraining, and providing feedback on AI agent behavior.

@progress/observability

AI agent instrumentation for TypeScript/JavaScript applications

tooltape

VCR for LLM agent loops — record tool calls once, replay deterministically in tests

@saishettar/iris-otel

OpenTelemetry-native instrumentation SDK for LLM/agent calls (Iris) - TypeScript

@alexvegman/toolsmith-core

Canonical IR types (ToolSpec, ToolSurface, Finding) for Toolsmith, a tool-surface linter for LLM agent tool definitions — see @alexvegman/toolsmith-engine.

jev-evals

Rubric-based eval harness for LLM/agent outputs, backed by the typesafe-ai/jev evaluation model via Vercel AI Gateway. Cheap enough (~$0.04/1M input tokens, one round trip per case) to run on every PR.

@krishnadobhal/rewind-sdk-js

Record every boundary an LLM agent crosses: withRewind, the recorder, the redactor, and the stores.

dsh-llm-verifier

Configurable DSH-native LLM verifier with a Web settings page

agentfootprint-lens

See through your agent's decisions. React components for debugging agentfootprint runs — messages, prompts, tool calls, decision scope, cost.

langfuse-core

NOT the latest Langfuse JS/TS SDK. Legacy v3 shared core — do not install directly. Current SDK: npm install @langfuse/tracing @langfuse/otel @opentelemetry/sdk-node

@moldea.ai/adapter-langchain

Deterministic runtime evidence and diagnostics for LangChain TypeScript agents.

@artemiskit/adapter-langchain

LangChain.js adapter for ArtemisKit - Test LangChain chains and agents

@matrixverify/verify

Fail-silent OpenTelemetry tracing SDK for Matrix claim verification

@agenticrun/sdk

Agenticrun client and LangChain integration for managed agent configuration and tracing

@arizeai/openinference-instrumentation-langchain-v0

OpenInference Instrumentation for LangChain.js Version 0.X

humanish

Open-source-safe CLI for persona simulation, observer review, and public-safe feedback drafts.

@moldea.ai/adapter-openai-agents-sdk

Deterministic runtime evidence and diagnostics for direct OpenAI Agents SDK integrations.

moda-ai

Official Node.js/TypeScript SDK for Moda — the continual learning layer for AI agents. Automatic conversation threading and OpenTelemetry-native tracing.

@moldea.ai/adapter-openai

Deterministic runtime evidence and diagnostics for direct OpenAI SDK integrations.

sht-portfolio-v2

Official TypeScript & JavaScript SDK and CLI for Shivanshu Tiwari Portfolio APIs, projects, and autonomous agent evaluation

langfuse-vercel

LEGACY Langfuse v3 integration for Vercel AI SDK (maintenance mode). For new projects use @langfuse/otel with the AI SDK's experimental_telemetry: npm install @langfuse/otel @langfuse/tracing

@artemiskit/adapter-deepagents

DeepAgents.js adapter for ArtemisKit - Test DeepAgents multi-agent systems

More in Agent frameworks & infrastructure