Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 55 of 57
Record and inspect AI agent API traffic and activity
Reconstructs from transcripts why an AI agent reached for protected data
Realtime AI Agent Office & Activity Visualizer for Antigravity, Claude Code, OpenAI Codex & AI CLIs
Record Playwright-driven AI agent runs with Steplight.
agent-harness observability: capture LLM/agent activity as OpenTelemetry spans with canonical gen_ai.* attributes and export over OTLP/HTTP JSON
A zero-install viewer for LLM agent traces: drop an OpenTelemetry export, see every model call, tool call, token and dollar on a timeline.
Client library for the Trellar confidence evaluation API
LangChain.js LLM cost tracking with provider-reported usage, scheduled pricing refresh, local memory or Redis storage.
Aigie, the JS/TS SDK for Kytte: the autonomous remediation platform for AI agents, evaluating every agent step in real time
AgentOS TypeScript SDK: run hosted agents, and report runs, events and spans from your own.
A flight recorder for AI agents: record your Anthropic or OpenAI agent to a hash-chained cassette, replay it offline, fork it with one tool result changed, and diff to the first divergence.
AI agent observability for Node.js: trace LLM calls, agents, errors, cost, evaluations and prompts
Vercel AI SDK adapter for the Astro messaging service. Wraps an Agent and auto-wires OpenTelemetry tracing.
Self-hosted control plane for budget-guarded AI agents.
Developer web console and diagnostic chamber for AI Oh-My-Pi — Elysia + Preact on Bun.
AI development tools for debugging and monitoring AI SDKs streams
SwarmWatch Universal Observability, Zero-Token Shannon Entropy Circuit Breaker & Safe Time-Machine for Multi-Agent LLM Systems in TypeScript & Node.js.
Efferent composable agent framework, 0.22.0. Node 24.
Efferent composable agent framework, 0.4.0. Bun on Linux.
Library for Agentic Engineering Evals
A standalone evaluation runner for agentic coding capabilities
Aforo AI Agent Metering SDK — instrument agent runs (sessions, steps, tool calls) for billing and analytics
Neens pre-prod evaluation runner — the CI one-liner (PULL/PUSH) with gate-as-code, for TypeScript/Node CI. TypeScript parity port of the Python neens-eval SDK.
Run Optimus agent tests against your agent, in your own process.
Live 3D views of agent systems (LangGraph, deepagents, any OpenTelemetry-instrumented agents) as a React component.
Capture and audit every AI agent decision. Auto-instruments the Vercel AI SDK; one-line wrappers for the OpenAI and Anthropic clients (@runback/sdk/openai, @runback/sdk/anthropic) and a LangGraph/LangChain callback handler (@runback/sdk/langgraph); a gene
One-line verifiable execution receipts for AutoGen swarms. Free, no API key, offline verification, chain-head anchoring to Nostr and Bitcoin.
One-line verifiable execution receipts for CrewAI crews. Free, no API key, offline verification, chain-head anchoring to Nostr and Bitcoin.
One-line verifiable execution receipts for Haystack pipelines. Free, no API key, offline verification, chain-head anchoring to Nostr and Bitcoin.
One-line verifiable execution receipts for LangChain agents. Free, no API key, offline verification, chain-head anchoring to Nostr and Bitcoin.
AER-1 verifiable workflow receipts for LangGraph multi-agent swarms
One-line verifiable execution receipts for LlamaIndex agents. Free, no API key, offline verification, chain-head anchoring to Nostr and Bitcoin.
One-line verifiable execution receipts for the OpenAI Agents SDK. Free, no API key, offline verification, chain-head anchoring to Nostr and Bitcoin.
One-line verifiable execution receipts for HuggingFace SmolAgents. Free, no API key, offline verification, chain-head anchoring to Nostr and Bitcoin.
Aforo AI Agent Metering SDK — instrument agent runs (sessions, steps, capability calls) for billing and analytics
Budget caps, runaway-loop detection and circuit breakers for AI agents. Zero dependencies, framework agnostic.
Causal capture SDK for branching multi-agent systems
Causal capture SDK for branching multi-agent systems
Chaos engineering and self-healing recovery patterns for multi-agent LLM systems: inject hallucination loops, tool failures, cyclic dependencies and stale memory, then measure how the system degrades and recovers.
Check AI agent success claims against trace evidence instead of trusting the final message.
Agent Control Panel SDK for Python: monitor AI agents in production and pause, resume, stop or swap their model without redeploying
Report token usage and estimated USD cost of AI agent sessions from local harness logs
Proof that AI agents' work really happened: re-runs your checks in a clean copy the agent can't edit, and writes a sealed receipt.
Check retained agent-evidence packets, histories and native reports offline
Versioned terms for claims about agent execution evidence
Live environmental footprint (energy, water, carbon) of AI coding agents: Claude Code, Codex CLI, Gemini CLI, OpenCode, Mistral Vibe and GitHub Copilot, computed with EcoLogits.
Process-metadata labels for OpenTelemetry agent spans
Ingest coding-agent session logs into DuckDB and query them read-only.
Notice when an unattended AI agent is stuck, pause it, and show why.
Analyze Claude Code, Cursor, OpenCode, CodeAgent, CodeArts, ICode, Codex CLI, Pi, and DeepSeek Harness sessions and saved Claude Code OpenTelemetry captures. Export metrics and charts as JSON or explore them in optional dashboards.
Evidence-first regression and contract testing for Gemini and Vertex AI agents.
A unified evaluation framework for AI agents, with a Python SDK and CLI
Measure how data-quality defects affect ML models and LLM agents, and validate data per consumer
Scan AI infrastructure against policy-as-code rule packs, evaluate AI agents with standard exams, and issue signed certificates when both pass.
AgentFeedback native CLI
Agent FinOps: cost attribution and ROI for AI agent spend
Fork your LLM agent at any step and sweep prompts, models or tool results to find what fixes it.
Live 3D views of agent systems, driven only by OpenTelemetry spans
Agent observability MCP server for querying OpenTelemetry traces, failures, tool health, security evidence, and regressions.
Typed synchronous and asynchronous Python access to the AgentHealth CLI
Agentic inference benchmark: closed-loop replay of real agent trajectories against any OpenAI-compatible server
A generic, plug-and-play Agentic Control Plane for MLOps Governance
Multi-Agent Observability & Auto-Recovery
Send your AI agent's production traces to AgentLasso: continuous behavioral coverage for AI agents.
AI-powered observability SDK for AI agents
Lightweight AgentLoop offline experiment SDK
FOLTL constraint verification for LLM agent traces
DeepEval experiment publishing for the Grafana Agent Observability Python SDK
Harbor live evaluation and job import for the Grafana Agent Observability Python SDK
Grafana Agent Observability plugin for Hermes Agent. Records LLM calls and tool executions as generations and OTel traces+metrics.
Measure how well a computer serves AI coding agents - and explain why
Load public agent-trajectory datasets and agent traces into one run record based on ATIF v1.7.
Shared, versioned domain primitives (Trace, Span, Envelope) for the AgentPlane products.
Evaluation and regression testing for AI agents
Analyse AI coding agent sessions: call trees, timings, tokens, cost and waste.
AgentPulse for Python: detect AI agents, negotiate Markdown and declare Content Signals (FastAPI/Starlette).
AI agent observability SDK — track runs, LLM calls, tool calls, and errors.
Local agent activity indicators for RGB devices
Show Antigravity CLI, Claude Code, Codex and Grok Build sessions and their subagents as a live tree: model, effort, context use, elapsed time, status.
See what every AI agent in your business is working on. The official Python SDK for Agents28.
AgentScope Service SDK: connect agent frameworks and report to the Control Plane
Execution traces for agent behavior - OpenTelemetry-style observability for AI agent workflows
A private, local dashboard for AI coding agent usage
Record AI-agent runs, replay them against a new version with every tool answered from the recording, and get a PASS/FAIL verdict.
SemiAnalysis AgentX inference benchmark harness, built on AIPerf
Zero-instrumentation, fully local X-ray for coding-agent sessions: interactive timeline, tool-call tree, token/cost curves and failure highlights in a single shareable HTML file.
Certify an AI agent's violation rate with an exact one-sided bound, and re-check a stricter safety gate against the logs you already have.
Shared provider contract suite and fakes for AI LLM Kernel.
Local developer telemetry agent for Antigravity AI usage and engineering signals
Graph and execution monitoring for LangGraph workflows
The content-free telemetry spool and its uploader for AirPrompter agents: minute windows as append-only segments (or in memory on serverless), the host budget, and direct-to-object-store upload under a grant.
LiteLLM usage adapter for AUDR (experimental)
Ask which benchmarks researchers use, how usage changes, and which new benchmarks are adopted
Efficient training data attribution (TDA) infrastructure for LLM-scale models.
Command-line tool for DriftWatch, a drift monitor for LLM apps.
The semantic layer for AI-agent evaluation: the Evaluation Intelligence Ontology (EIO) and the Portable Evaluation Record (PER 2.1). Convert, validate, verify, explain.
EQTY lineage callback handler for DeepAgents.
EQTY lineage callback handler for LangChain and LangGraph.
CI/CD evaluation infrastructure for AI agents
Agent cost regression CI: tells you which change in a pull request made your LLM agent more expensive, and by how much.