Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 53 of 57
Deterministic runtime evidence and diagnostics for LangGraph TypeScript workflows.
Statistical evaluation and reporting for LLM-based software engineering systems.
Reference emitter for the agent-audit OpenTelemetry semantic convention: proposed -> decided -> executed.
LLM agent execution tracer with failure attribution
Discover local Claude Code / Codex session logs and export them as OpenTelemetry GenAI-compatible traces.
Automatic LangChain Agent observability for AgentBase AgentOps
Unified observability for LLM/text, image, and video calls (Django adapter)
Reliability and audit-ready testing for LLM agents (Python bridge to the AgentEval engine).
Normalization layer for OpenTelemetry GenAI agent traces. Single parsing pass, framework detection, span classification with confidence scores.
Peer review for AI agents using independent model juries and deterministic aggregation.
Behavioural monitoring for AI agents: runtime baselines, scope enforcement and cold-start-safe anomaly detection.
Append-only JSONL session traces, assemble, and MCP inspect tools
ARK Runtime SDK — attach ARK decision telemetry, cost attribution, and (experimental) constrained supervision around your existing agent.
Thin HTTP client for the RAG debugging/eval platform — register an external RAG, upload datasets, run experiments, fetch results.
A first-principles evaluation framework for LLM agents based on causal invariants, state transitions, and counterfactuals.
Cost and prompt-cache diagnostics for Claude Code session transcripts
Measure cumulative Codex and Claude Code agent runtime without decoding conversation content.
Anthropic target and judges for digline: prompt file in, priced Response out.
Amazon Bedrock target and judges for digline, on the Converse API.
OpenAI target and judges for digline, at any OpenAI-compatible endpoint.
Open-source framework for building agent configurations, testing AI agents, and comparing reproducible AI-system observations.
LangGraph tracing for Databricks MLflow
Put one question to several LLMs independently and read every answer in full -- parallel judges, failure classes, cost, and an anonymised rebuttal round
Evaluation pipeline for LLM watermark detection experiments
Practical local LLM evaluation on real hardware
Server-side SDK for tracking LLM token usage, costs, and observability data in LLMtrack
Library for building agent tests using Microsoft Agents SDK
Fake AG-UI agent backend for testing. Deterministic scripted event streams, session record & replay, failure injection. No LLM needed.
Regression tests for AI agents. Record once, replay offline, catch behavioural regressions.
Runtime-scoped LLM hooks for verified Python handlers and source patches
RAG evaluation toolkit with readable LLM-judge prompts and local-first (Ollama) support.
RAG evaluation and root-cause diagnosis SDK/CLI
The complete RAG evaluation library. Scores, diagnoses, and domain-aware.
Haystack integration for rhesis
Schema definitions and utilities for the AgentLogs dataset
Official Python Telemetry & Monitoring Client for Sannex AI Operations
Tencent Cloud CLS observability for AgentScope agents
CLS observability SDK for OpenAI Agents SDK — per-turn trace model, direct upload to Tencent Cloud CLS
OpenTelemetry-first LLM tracing SDK with OpenAI instrumentation
Claude Code hook instrumentation plugin for TruLens.
Library to systematically track and evaluate LLM based applications.
WASM Node.js bindings for libdatadog
Convert agent session logs (Claude Code JSONL) into Perfetto/Chrome trace JSON for ui.perfetto.dev
AD-001..AD-015: the agentic-dataset governance contract as language-neutral executable conformance vectors, plus a runner that needs no access to an implementation's internals
Framework-agnostic observability, debugging, and deterministic issue detection for AI agents.
Ragas-inspired RAG quality evaluation, synthetic dataset generation, and continuous runtime observability for Agno.
The testing framework for AI agents — think pytest, but for agents.
Regression testing for LLM applications (CI for prompts)
Emit a Testimony Record from a LangGraph human-in-the-loop flow, so the record can say who approved.
Deterministic semantic record and replay for LLM agent tests
Anchor your LangChain.js agent's run tree as it executes: one callback handler turns every chain, LLM and tool step into a signed evidence envelope, sent to Sanning and witnessed on an interval to a permanent public record. Prompts and outputs are hashed
OpenAI Node SDK adapter for the GraphMind live agent debugger: wrap your OpenAI client and tools to stream execution events and honour pause / step / inject / retry / abort gates.
Mock BuilderClient for @chatu-ai/builder-sdk: scripted event replay with latency & disconnect injection
Vercel AI SDK integration package for MLflow Tracing
Customer-controlled OpenTelemetry capture SDK for reconstructing AI agent activity.
Multi-step agent safety compliance benchmarking — finds the critical depth D* where LLM agents fail to maintain safety policies across sequential interactions.
A lightweight, in-process circuit breaker for LLM token and cost budgets. Part of the agentbelt suite.
Detect repeat, cyclic, and stagnation loops in AI agent tool-call sequences, with cross-session pattern learning. Part of the agentbelt suite.
Agent-aware span sampling: keeps errors, cost/latency outliers, retries, and novel calls; samples the rest. Part of the agentbelt suite.
Budget circuit breaking, loop detection, and span sampling for AI agent workflows, in one install.
Multi-agent audit CLI: cost governance, decision audit, evidence chain, shadow agent detection, compliance
AgentShip observability — the OpenTelemetry implementation of the Observer port: span-tree exporter pipeline (Phoenix/Langfuse/LangSmith), the one-per-process LiteLLM cost/token callback, and the studio launcher. The vendor-free contract lives in agentship-core.
LangChain integration for Anlyon: actions as tools, and chains and agents traced as Anlyon runs
LlamaIndex integration for Anlyon: actions as tools, plus a vector store on the memory namespace
OpenAI Agents SDK 的腾讯云 CLS Agent 可观测接入 —— 一行初始化,无需手动埋点
Debug LangGraph-based agents visually - live execution graph, node inspection, and run replay, all from a single line.
Language-agnostic LLM cost metering via HTTP proxy — track spend per customer and feature
Standardized methodology and dataset for evaluating LLM-based digital forensic timeline analysis
FSM syntactic engine to repair truncated/broken JSON from LLMs (streaming, Pydantic, zero-regex core)
Evidence-based reliability analyzer and root-cause diagnosis engine for LLM, RAG, and Agent systems.
A pytest plugin for regression-testing LLM applications: metamorphic relations and flakiness statistics.
Observability and drift detection for multi-agent systems (AutoGen, LangChain/LangGraph)
Agent Execution Evidence for Pydantic AI, by Sentience Governor
An open source observability RAG framework.
CLS observability SDK for LangChain/LangGraph — 6-layer span hierarchy, direct upload to Tencent Cloud CLS
Fail-open reliability monitoring for unattended agent jobs
Discover, normalize, and understand AI-agent histories locally.
Universal OpenTelemetry API bridge for correlated ai-agent-sdk spans, metrics, and logs
A cross-agent-runtime unified event and policy layer for observing, constraining, and providing feedback on AI agent behavior.
AI agent instrumentation for TypeScript/JavaScript applications
VCR for LLM agent loops — record tool calls once, replay deterministically in tests
OpenTelemetry-native instrumentation SDK for LLM/agent calls (Iris) - TypeScript
Canonical IR types (ToolSpec, ToolSurface, Finding) for Toolsmith, a tool-surface linter for LLM agent tool definitions — see @alexvegman/toolsmith-engine.
Rubric-based eval harness for LLM/agent outputs, backed by the typesafe-ai/jev evaluation model via Vercel AI Gateway. Cheap enough (~$0.04/1M input tokens, one round trip per case) to run on every PR.
Record every boundary an LLM agent crosses: withRewind, the recorder, the redactor, and the stores.
Configurable DSH-native LLM verifier with a Web settings page
See through your agent's decisions. React components for debugging agentfootprint runs — messages, prompts, tool calls, decision scope, cost.
NOT the latest Langfuse JS/TS SDK. Legacy v3 shared core — do not install directly. Current SDK: npm install @langfuse/tracing @langfuse/otel @opentelemetry/sdk-node
Deterministic runtime evidence and diagnostics for LangChain TypeScript agents.
LangChain.js adapter for ArtemisKit - Test LangChain chains and agents
Fail-silent OpenTelemetry tracing SDK for Matrix claim verification
Agenticrun client and LangChain integration for managed agent configuration and tracing
OpenInference Instrumentation for LangChain.js Version 0.X
Open-source-safe CLI for persona simulation, observer review, and public-safe feedback drafts.
Deterministic runtime evidence and diagnostics for direct OpenAI Agents SDK integrations.
Official Node.js/TypeScript SDK for Moda — the continual learning layer for AI agents. Automatic conversation threading and OpenTelemetry-native tracing.
Deterministic runtime evidence and diagnostics for direct OpenAI SDK integrations.
Official TypeScript & JavaScript SDK and CLI for Shivanshu Tiwari Portfolio APIs, projects, and autonomous agent evaluation
LEGACY Langfuse v3 integration for Vercel AI SDK (maintenance mode). For new projects use @langfuse/otel with the AI SDK's experimental_telemetry: npm install @langfuse/otel @langfuse/tracing
DeepAgents.js adapter for ArtemisKit - Test DeepAgents multi-agent systems