Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 55 of 57

agenticreplay

Record and inspect AI agent API traffic and activity

@agentwhy/cli

Reconstructs from transcripts why an AI agent reached for protected data

agmon

Realtime AI Agent Office & Activity Visualizer for Antigravity, Claude Code, OpenAI Codex & AI CLIs

@steplight/sdk

Record Playwright-driven AI agent runs with Steplight.

@hachibox/genai-otel

agent-harness observability: capture LLM/agent activity as OpenTelemetry spans with canonical gen_ai.* attributes and export over OTLP/HTTP JSON

@antonsoloviev/tracelens

A zero-install viewer for LLM agent traces: drop an OpenTelemetry export, see every model call, tool call, token and dollar on a timeline.

trellar

Client library for the Trellar confidence evaluation API

langchain-cost-tracker

LangChain.js LLM cost tracking with provider-reported usage, scheduled pricing refresh, local memory or Redis storage.

kytte

Aigie, the JS/TS SDK for Kytte: the autonomous remediation platform for AI agents, evaluating every agent step in real time

@agentos-sdk/core

AgentOS TypeScript SDK: run hosted agents, and report runs, events and spans from your own.

@ardaulas/blackbox

A flight recorder for AI agents: record your Anthropic or OpenAI agent to a hash-chained cassette, replay it offline, fork it with one tool result changed, and diff to the first divergence.

risicare

AI agent observability for Node.js: trace LLM calls, agents, errors, cost, evaluations and prompts

@astropods/adapter-ai-sdk

Vercel AI SDK adapter for the Astro messaging service. Wraps an Agent and auto-wires OpenTelemetry tracing.

@wardby/cli

Self-hosted control plane for budget-guarded AI agents.

ompchamber

Developer web console and diagnostic chamber for AI Oh-My-Pi — Elysia + Preact on Bun.

ai-sdk-devtools

AI development tools for debugging and monitoring AI SDKs streams

@swarmwatch/sdk

SwarmWatch Universal Observability, Zero-Token Shannon Entropy Circuit Breaker & Safe Time-Machine for Multi-Agent LLM Systems in TypeScript & Node.js.

@xandreed/evals

Efferent composable agent framework, 0.22.0. Node 24.

@xandreed/plugin-telemetry

Efferent composable agent framework, 0.4.0. Bun on Linux.

agentic-evals

Library for Agentic Engineering Evals

@bjoernrochel/sevro

A standalone evaluation runner for agentic coding capabilities

@aforoai/agent-metering

Aforo AI Agent Metering SDK — instrument agent runs (sessions, steps, tool calls) for billing and analytics

neens-eval

Neens pre-prod evaluation runner — the CI one-liner (PULL/PUSH) with gate-as-code, for TypeScript/Node CI. TypeScript parity port of the Python neens-eval SDK.

@optimus-ai/sdk

Run Optimus agent tests against your agent, in your own process.

agentglow

Live 3D views of agent systems (LangGraph, deepagents, any OpenTelemetry-instrumented agents) as a React component.

@runback/sdk

Capture and audit every AI agent decision. Auto-instruments the Vercel AI SDK; one-line wrappers for the OpenAI and Anthropic clients (@runback/sdk/openai, @runback/sdk/anthropic) and a LangGraph/LangChain callback handler (@runback/sdk/langgraph); a gene

aer1-autogen

One-line verifiable execution receipts for AutoGen swarms. Free, no API key, offline verification, chain-head anchoring to Nostr and Bitcoin.

aer1-crewai

One-line verifiable execution receipts for CrewAI crews. Free, no API key, offline verification, chain-head anchoring to Nostr and Bitcoin.

aer1-haystack

One-line verifiable execution receipts for Haystack pipelines. Free, no API key, offline verification, chain-head anchoring to Nostr and Bitcoin.

aer1-langchain

One-line verifiable execution receipts for LangChain agents. Free, no API key, offline verification, chain-head anchoring to Nostr and Bitcoin.

aer1-langgraph

AER-1 verifiable workflow receipts for LangGraph multi-agent swarms

aer1-llamaindex

One-line verifiable execution receipts for LlamaIndex agents. Free, no API key, offline verification, chain-head anchoring to Nostr and Bitcoin.

aer1-openai-agents

One-line verifiable execution receipts for the OpenAI Agents SDK. Free, no API key, offline verification, chain-head anchoring to Nostr and Bitcoin.

aer1-smolagents

One-line verifiable execution receipts for HuggingFace SmolAgents. Free, no API key, offline verification, chain-head anchoring to Nostr and Bitcoin.

aforo-agent-metering

Aforo AI Agent Metering SDK — instrument agent runs (sessions, steps, capability calls) for billing and analytics

agent-budget-guard-py

Budget caps, runaway-loop detection and circuit breakers for AI agents. Zero dependencies, framework agnostic.

agent-casuality

Causal capture SDK for branching multi-agent systems

agent-causality

Causal capture SDK for branching multi-agent systems

agent-chaos-engineering

Chaos engineering and self-healing recovery patterns for multi-agent LLM systems: inject hallucination loops, tool failures, cyclic dependencies and stale memory, then measure how the system degrades and recovers.

agent-claimcheck

Check AI agent success claims against trace evidence instead of trusting the final message.

agent-control-panel

Agent Control Panel SDK for Python: monitor AI agents in production and pause, resume, stop or swap their model without redeploying

agent-cost-estimator

Report token usage and estimated USD cost of AI agent sessions from local harness logs

agent-countercheck

Proof that AI agents' work really happened: re-runs your checks in a clean copy the agent can't edit, and writes a sealed receipt.

agent-evidence-observer

Check retained agent-evidence packets, histories and native reports offline

agent-evidence-vocabulary

Versioned terms for claims about agent execution evidence

agent-footprint

Live environmental footprint (energy, water, carbon) of AI coding agents: Claude Code, Codex CLI, Gemini CLI, OpenCode, Mistral Vibe and GitHub Copilot, computed with EcoLogits.

agent-reliability-otel-labels

Process-metadata labels for OpenTelemetry agent spans

agent-session-analytics

Ingest coding-agent session logs into DuckDB and query them read-only.

agent-supra

Notice when an unattended AI agent is stuck, pause it, and show why.

agent-trajviz

Analyze Claude Code, Cursor, OpenCode, CodeAgent, CodeArts, ICode, Codex CLI, Pi, and DeepSeek Harness sessions and saved Claude Code OpenTelemetry captures. Export metrics and charts as JSON or explore them in optional dashboards.

agentaxiom

Evidence-first regression and contract testing for Gemini and Vertex AI agents.

agentcompass

A unified evaluation framework for AI agents, with a Python SDK and CLI

agentdq

Measure how data-quality defects affect ML models and LLM agents, and validate data per consumer

agenteval-toolkit

Scan AI infrastructure against policy-as-code rule packs, evaluate AI agents with standard exams, and issue signed certificates when both pass.

agentfeedback-cli

AgentFeedback native CLI

agentfin

Agent FinOps: cost attribution and ROI for AI agent spend

agentfork-cli

Fork your LLM agent at any step and sweep prompts, models or tool results to find what fixes it.

agentglow

Live 3D views of agent systems, driven only by OpenTelemetry spans

agenthawk

Agent observability MCP server for querying OpenTelemetry traces, failures, tool health, security evidence, and regressions.

agenthealth

Typed synchronous and asynchronous Python access to the AgentHealth CLI

agentic-bench

Agentic inference benchmark: closed-loop replay of real agent trajectories against any OpenAI-compatible server

agentic-ops

A generic, plug-and-play Agentic Control Plane for MLOps Governance

agenticu

Multi-Agent Observability & Auto-Recovery

agentlasso

Send your AI agent's production traces to AgentLasso: continuous behavioral coverage for AI agents.

agentlensx

AI-powered observability SDK for AI agents

agentloop-experiment-sdk

Lightweight AgentLoop offline experiment SDK

agentltl

FOLTL constraint verification for LLM agent traces

agento11y-deepeval

DeepEval experiment publishing for the Grafana Agent Observability Python SDK

agento11y-harbor

Harbor live evaluation and job import for the Grafana Agent Observability Python SDK

agento11y-hermes

Grafana Agent Observability plugin for Hermes Agent. Records LLM calls and tool executions as generations and OTel traces+metrics.

agentperf

Measure how well a computer serves AI coding agents - and explain why

agentpile

Load public agent-trajectory datasets and agent traces into one run record based on ATIF v1.7.

agentplane-primitives

Shared, versioned domain primitives (Trace, Span, Envelope) for the AgentPlane products.

agentprobe-eval

Evaluation and regression testing for AI agents

agentprof

Analyse AI coding agent sessions: call trees, timings, tokens, cost and waste.

agentpulse-asgi

AgentPulse for Python: detect AI agents, negotiate Markdown and declare Content Signals (FastAPI/Starlette).

agentradar-sdk

AI agent observability SDK — track runs, LLM calls, tool calls, and errors.

agentrgb

Local agent activity indicators for RGB devices

agents-tree

Show Antigravity CLI, Claude Code, Codex and Grok Build sessions and their subagents as a live tree: model, effort, context use, elapsed time, status.

agents28

See what every AI agent in your business is working on. The official Python SDK for Agents28.

agentscope-service-sdk

AgentScope Service SDK: connect agent frameworks and report to the Control Plane

agentsec-agentwatch

Execution traces for agent behavior - OpenTelemetry-style observability for AI agent workflows

agenttelemetry-app

A private, local dashboard for AI coding agent usage

agenttrace-vcr

Record AI-agent runs, replay them against a new version with every tool answered from the recording, and get a PASS/FAIL verdict.

agentx-harness

SemiAnalysis AgentX inference benchmark harness, built on AIPerf

agentxray-replay

Zero-instrumentation, fully local X-ray for coding-agent sessions: interactive timeline, tool-call tree, token/cost curves and failure highlights in a single shareable HTML file.

ai-agent-risk

Certify an AI agent's violation rate with an exact one-sided bound, and re-check a stricter safety gate against the logs you already have.

ai-llm-kernel-testing

Shared provider contract suite and fakes for AI LLM Kernel.

ai-telemetry-agent

Local developer telemetry agent for Antigravity AI usage and engineering signals

aictrl-langgraph

Graph and execution monitoring for LangGraph workflows

airprompter-agent-telemetry

The content-free telemetry spool and its uploader for AirPrompter agents: minute windows as append-only segments (or in memory on serverless), the host budget, and direct-to-object-store upload under a grant.

audr-adapter-litellm

LiteLLM usage adapter for AUDR (experimental)

bench2agent

Ask which benchmarks researchers use, how usage changes, and which new benchmarks are adopted

dattri-llm

Efficient training data attribution (TDA) infrastructure for LLM-scale models.

driftwatch-llm

Command-line tool for DriftWatch, a drift monitor for LLM apps.

eio-agents

The semantic layer for AI-agent evaluation: the Evaluation Intelligence Ontology (EIO) and the Portable Evaluation Record (PER 2.1). Convert, validate, verify, explain.

eqty-lineage-deepagents

EQTY lineage callback handler for DeepAgents.

eqty-lineage-langchain

EQTY lineage callback handler for LangChain and LangGraph.

evalforge-agents

CI/CD evaluation infrastructure for AI agents

ezop-agentcost

Agent cost regression CI: tells you which change in a pull request made your LLM agent more expensive, and by how much.

More in Agent frameworks & infrastructure