Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 49 of 57
A LangChain callback handler that emits Agent Arc Status Protocol events at chain/agent boundaries.
Monitor your AI agents in 3 lines of code — tokens, cost, latency, and silent failures
LangSmith tracing for skein-js — run identity and thread grouping on every graph call.
LangChain/LangGraph callback adapter feeding graph-trace.
PostHog analytics for skein-js — run lifecycle events plus $ai_generation LLM analytics.
Structured tracing model for graph agent runs.
Point it at your coding-agent session traces (Claude Code, Codex, OpenCode, Gemini, Pi, …) and get failure-mode + efficiency findings. CLI + SDK over the @tangle-network/agent-eval analyst suite — observe live sessions, run your own analysts, redact, and
Telemetry collector for AI agents — Node.js / TypeScript port. Streams every run, LLM call, tool invocation, and chain step from LangChain.js / Mastra / Vercel AI SDK / Google GenAI (Gemini) — or a manual API — to the Dobby AI Control Plane for governance
OpenBox LangChain adapter for TypeScript — governance + observability parity over @openbox-ai/openbox-sdk-ts for LangChain JS/TS agents
Evaluate and improve AI agents from runs, traces, judges, and feedback. Compare candidates, cluster failures, measure lift, and gate releases.
Eden TypeScript runtime SDK — auto-instrumentation for OpenAI, Anthropic, Vercel AI, LangChain, Mastra, Bedrock, and Gemini with fire-and-forget telemetry to the Eden gateway.
Olympus dashboard for monitoring OpenClaw workspaces, agents, lineage, configuration, cron activity, and costs.
Network-tab-style DevTools for LLM calls in Angular apps. See every prompt, response, token, and dollar your app spends — without leaving the browser.
OpenInference instrumentation for OpenAI Agents SDK
Free local OSS premortem, self-heal, and no-spend Golden Loop readiness CLI for plans, launches, and installable AI agent repositories.
OpenAI integration package for MLflow Tracing
autocontext — always-on agent evaluation harness
Declarative orchestration + observability for autonomous agent loops on the Claude Agent SDK
AI assistant status monitor
Vibe Coding Tracker - AI coding assistant telemetry/usage parser, aggregate JSONL events into CodeAnalysis results
A library for running evaluations for AI use cases
Optional OpenTelemetry adapter for @open-multi-agent/core TraceRecord v2.
Read-only TUI to watch Claude Code multi-agent workflows live
Real-time dashboard for OpenCode multi-agent activity, sessions, status, and tool usage
Aroha distributed tracing — W3C Trace Context propagation for multi-agent sagas
Respan instrumentation plugin for the Eve agent framework
Eve (Vercel agent framework) telemetry integration for telemetry.dev.
> Local tracing & observability devtool for the Rejelly agent framework — inspect traces, spans, tool calls, and token usage.
Vendor-neutral, evidence-backed readiness benchmark for agentic software development harnesses.
Hash-chained evidence ledger recording what each ADLC gate verified — agentic provenance with optional HMAC signing (C11).
Evaluation framework and CLI for WebMCP tools and agentic workflows
Crooked Pattern — Agentic Readiness Score (ARS) scanner, CLI + MCP server
effigent — the Optimizer CLI: capture AI-agent runs (Claude Code, Codex, any OTel agent) and send them to your Optimizer workspace.
The ACP evaluator SDK — the fleet-reuse artifact. Any agent embeds it to fetch → verify → cache → evaluate → seal a governance bundle LOCALLY. Framework-agnostic. Fail-static.
One dark dashboard to find and open every agent/AI dashboard running on your localhost. Zero dependencies, runs with npx.
AgentBOM AutoGen adapter
The circuit breaker for AI agents. Detect runaway loops and cost spikes in real time.
Auto-detecting CLI installer & circuit breaker configuration generator for AI Agents
The Context Company CLI — zero-friction AI agent observability setup
UserVane frontend capture for CopilotKit (deterministic turn-resolution feedback)
Static, evidence-backed health checks for AI agent projects
LedgeLM evaluation helpers for LangGraph graphs and nodes
CLI to inspect LangGraph agent trajectories locally - zero infra, localhost only
AG-UI lifecycle and concurrency checker for LangGraph agents
Event schema, LangGraph adapter, and approval helpers for Agent Action Inspector
Normalize native agent transcripts into a shared trajectory format.
Local-first Engram Studio CLI.
OpenTelemetry spans and metrics for skein-js runs — works with any OTLP backend.
Record-and-replay for agent decision graphs: capture what an agent did, save it to the control plane, and replay a request with the downstream calls stubbed.
Per-stage token/cost attribution and silent-degradation detection for multi-agent workflow runs (stdlib-only).
Blame analysis for multi-agent runs: read an OTEL trace, name where quality broke
Two-tier detection/scoring/blame pipeline over Redis Streams (build spec section 4)
A framework-agnostic evaluation harness for tool-using AI agents. Define scenario packs, run agents consistently, score outcomes and trajectories, and gate releases with evidence.
git blame for AI runs
Post-hoc log-analysis dashboard for AI agent transcripts (Claude Code JSONL). Reads session logs to spot order-prompt / tool-call mismatches; a log-reading tool for a human maintainer, not an anomaly monitor.
AI agent observability with deterministic record/replay — capture any LangGraph or OpenAI Agents SDK run, replay it offline at zero API cost.
Framework-agnostic completion-lie detector for AI agents -- catches an agent turn whose final claim contradicts its own tool evidence.
Statistical regression testing for LLM agents. Get a p-value on whether behavior actually shifted between versions, not just whether one run looked different. Apache 2.0, self-hostable, no SaaS dependency -- a Promptfoo alternative for the statistical testing gap threshold-based eval tools don't cover.
Local-first policy and trajectory-regression harness for tool-using AI agents: deterministic trace validation, policy-as-code trajectory rules, baseline-vs-candidate regression gates, and CI-ready JSON/Markdown/JUnit/SARIF reports.
Record & replay LLM API calls for deterministic agent tests.
Record, replay, and time-travel-debug LLM agent runs.
Agent-RL Observability SDK — lightweight OpenTelemetry-based tracing
Versioned evaluation, coverage, and release governance for Agent Skills
A quality gate and spend auditor for your AI agent fleet.
CLI contract v1 conformance kit for the agent suite (Plan 018 WI-2 / Plan 019 B1).
agent-tco: Pareto-optimal configuration of agentic workflows.
Run-aware token governance for agents: budgets, policies and halts on Chronicle and the AgentPlane control plane
Normalize native agent transcripts into a shared trajectory format.
Walk-forward / out-of-sample validation for agent evals — detect eval-set overfitting the way quants detect backtest overfitting.
Renamed — this project continues as iiwi
Local-first Agent Work Intelligence for coding agents: usage truth, recorded work, and honest joins
Evaluation framework for multi-agent AI workflows: handoff quality scoring, failure attribution, loop detection, and cost guardrails.
Local CLI that profiles token spend for AI coding agent sessions (Claude, Cursor, Codex, Ollama).
Statistical regression harness for MCP tool descriptions (measures whether a change actually moved agent task success) + a deterministic defect linter as a secondary utility
Local-first safety and reliability evaluation framework for AI coding agents.
Predict LLM-agent-run failures in real time — and intervene before the tokens are spent
Independent, auditable verification for coding-agent claims
Standalone fault injection toolkit for LLM calls and agentic workflows, with an optional AgenticLens integration.
Evaluate agentic systems with reproducible, evidence-first grading.
In-memory observability database for LLM agents: logs tool calls, messages, thinking, responses, tokens, and carried-in context.
Token and cost tracking harness for AgenticStack: per-agent usage, USD estimates, and budgets
Zero-dependency, terminal-native Python middleware for resilient and observable AI agents.
A powerful observability and debugging SDK for AI Agents
A CI linter for AI agent execution traces.
OpenTelemetry-native observability for multi-agent LLM systems
A local, offline digest of your coding-agent sessions. Zero dependencies.
Profiler and optimization layer for AI agent execution graphs
Detect and kill infinite AI agent loops before they burn your API budget.
Monitor quota, tokens, and gateway balances for AI coding CLIs.
Lightweight SDK to track AI agent tool calls, costs, and latency.
Parses agent harness session logs into a unified event schema. Python wrapper around the agentminutes Go CLI.
Grafana Agent Observability Python SDK
Anthropic helper wrappers for the Grafana Agent Observability Python SDK
Claude Agent SDK hooks for the Grafana Agent Observability Python SDK
Gemini helper wrappers for the Grafana Agent Observability Python SDK
Google ADK callback handlers for the Grafana Agent Observability Python SDK
LangChain callback handlers for the Grafana Agent Observability Python SDK
LangGraph callback handlers for the Grafana Agent Observability Python SDK
LiteLLM callback handler for the Grafana Agent Observability Python SDK
LlamaIndex callback handlers for the Grafana Agent Observability Python SDK