Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 49 of 57

@agent-arc-status/adapter-langchain

A LangChain callback handler that emits Agent Arc Status Protocol events at chain/agent boundaries.

opsveritas-sdk

Monitor your AI agents in 3 lines of code — tokens, cost, latency, and silent failures

@skein-js/langsmith

LangSmith tracing for skein-js — run identity and thread grouping on every graph call.

@escouade/graph-trace-langchain

LangChain/LangGraph callback adapter feeding graph-trace.

@skein-js/posthog

PostHog analytics for skein-js — run lifecycle events plus $ai_generation LLM analytics.

@escouade/graph-trace

Structured tracing model for graph agent runs.

@tangle-network/traces

Point it at your coding-agent session traces (Claude Code, Codex, OpenCode, Gemini, Pi, …) and get failure-mode + efficiency findings. CLI + SDK over the @tangle-network/agent-eval analyst suite — observe live sessions, run your own analysts, redact, and

@dobbyai/collector

Telemetry collector for AI agents — Node.js / TypeScript port. Streams every run, LLM call, tool invocation, and chain step from LangChain.js / Mastra / Vercel AI SDK / Google GenAI (Gemini) — or a manual API — to the Dobby AI Control Plane for governance

@openbox-ai/openbox-langchain-governance

OpenBox LangChain adapter for TypeScript — governance + observability parity over @openbox-ai/openbox-sdk-ts for LangChain JS/TS agents

@tangle-network/agent-eval

Evaluate and improve AI agents from runs, traces, judges, and feedback. Compare candidates, cluster failures, measure lift, and gate releases.

@eden-ai/sdk

Eden TypeScript runtime SDK — auto-instrumentation for OpenAI, Anthropic, Vercel AI, LangChain, Mastra, Bedrock, and Gemini with fire-and-forget telemetry to the Eden gateway.

@flame0510/olympus

Olympus dashboard for monitoring OpenClaw workspaces, agents, lineage, configuration, cron activity, and costs.

ngx-ai-devtools

Network-tab-style DevTools for LLM calls in Angular apps. See every prompt, response, token, and dollar your app spends — without leaving the browser.

@edgeone/openinference-instrumentation-openai-agents

OpenInference instrumentation for OpenAI Agents SDK

agoragentic-premortem-golden-loop

Free local OSS premortem, self-heal, and no-spend Golden Loop readiness CLI for plans, launches, and installable AI agent repositories.

@mlflow/openai

OpenAI integration package for MLflow Tracing

autoctx

autocontext — always-on agent evaluation harness

looplens

Declarative orchestration + observability for autonomous agent loops on the Claude Agent SDK

vibemon

AI assistant status monitor

@mai0313/vibe-coding-tracker

Vibe Coding Tracker - AI coding assistant telemetry/usage parser, aggregate JSONL events into CodeAnalysis results

@arizeai/phoenix-evals

A library for running evaluations for AI use cases

@open-multi-agent/otel

Optional OpenTelemetry adapter for @open-multi-agent/core TraceRecord v2.

@pimlabs/agentop

Read-only TUI to watch Claude Code multi-agent workflows live

opencode-agent-graph

Real-time dashboard for OpenCode multi-agent activity, sessions, status, and tool usage

@aroha-sdk/telemetry

Aroha distributed tracing — W3C Trace Context propagation for multi-agent sagas

@respan/instrumentation-eve

Respan instrumentation plugin for the Eve agent framework

@telemetry-dev/eve

Eve (Vercel agent framework) telemetry integration for telemetry.dev.

@rejelly/devtool

> Local tracing & observability devtool for the Rejelly agent framework — inspect traces, spans, tool calls, and token usage.

agentic-scorecard

Vendor-neutral, evidence-backed readiness benchmark for agentic software development harnesses.

@adlc/gate-manifest

Hash-chained evidence ledger recording what each ADLC gate verified — agentic provenance with optional HMAC signing (C11).

webmcp-evals

Evaluation framework and CLI for WebMCP tools and agentic workflows

@grosspoetrysystems/crooked-pattern

Crooked Pattern — Agentic Readiness Score (ARS) scanner, CLI + MCP server

effigent

effigent — the Optimizer CLI: capture AI-agent runs (Claude Code, Codex, any OTel agent) and send them to your Optimizer workspace.

@getvda/evaluator-sdk

The ACP evaluator SDK — the fleet-reuse artifact. Any agent embeds it to fetch → verify → cache → evaluate → seal a governance bundle LOCALLY. Framework-agnostic. Fail-static.

dashscout

One dark dashboard to find and open every agent/AI dashboard running on your localhost. Zero dependencies, runs with npx.

@wasmagent/agentbom-autogen

AgentBOM AutoGen adapter

moven-sdk

The circuit breaker for AI agents. Detect runaway loops and cost spikes in real time.

moven-ai-cli

Auto-detecting CLI installer & circuit breaker configuration generator for AI Agents

@contextcompany/liftoff

The Context Company CLI — zero-friction AI agent observability setup

@uservane/copilotkit

UserVane frontend capture for CopilotKit (deterministic turn-resolution feedback)

agent-health

Static, evidence-backed health checks for AI agent projects

@ledgelm/langgraph

LedgeLM evaluation helpers for LangGraph graphs and nodes

@kashifmuhammad/agent-inspector

CLI to inspect LangGraph agent trajectories locally - zero infra, localhost only

@kiri_ikki/thread-agent-check

AG-UI lifecycle and concurrency checker for LangGraph agents

@kashifmuhammad/agent-inspector-core

Event schema, LangGraph adapter, and approval helpers for Agent Action Inspector

@letta-ai/trajectory

Normalize native agent transcripts into a shared trajectory format.

@engramviz/cli

Local-first Engram Studio CLI.

@skein-js/otel

OpenTelemetry spans and metrics for skein-js runs — works with any OTLP backend.

agent-chronicle

Record-and-replay for agent decision graphs: capture what an agent did, save it to the control plane, and replay a request with the downstream calls stubbed.

agent-cost-attribution

Per-stage token/cost attribution and silent-degradation detection for multi-agent workflow runs (stdlib-only).

agent-detective

Blame analysis for multi-agent runs: read an OTEL trace, name where quality broke

agent-detective-worker

Two-tier detection/scoring/blame pipeline over Redis Streams (build spec section 4)

agent-eval-forge

A framework-agnostic evaluation harness for tool-using AI agents. Define scenario packs, run agents consistently, score outcomes and trajectories, and gate releases with evidence.

agent-flight

git blame for AI runs

agent-log-verifier

Post-hoc log-analysis dashboard for AI agent transcripts (Claude Code JSONL). Reads session logs to spot order-prompt / tool-call mismatches; a log-reading tool for a human maintainer, not an anomaly monitor.

agent-observability-trace-cli

AI agent observability with deterministic record/replay — capture any LangGraph or OpenAI Agents SDK run, replay it offline at zero API cost.

agent-polygraph

Framework-agnostic completion-lie detector for AI agents -- catches an agent turn whose final claim contradicts its own tool evidence.

agent-regress-cli

Statistical regression testing for LLM agents. Get a p-value on whether behavior actually shifted between versions, not just whether one run looked different. Apache 2.0, self-hostable, no SaaS dependency -- a Promptfoo alternative for the statistical testing gap threshold-based eval tools don't cover.

agent-reliability-harness

Local-first policy and trajectory-regression harness for tool-using AI agents: deterministic trace validation, policy-as-code trajectory rules, baseline-vs-candidate regression gates, and CI-ready JSON/Markdown/JUnit/SARIF reports.

agent-replay-py

Record & replay LLM API calls for deterministic agent tests.

agent-rewind

Record, replay, and time-travel-debug LLM agent runs.

agent-rl-observability

Agent-RL Observability SDK — lightweight OpenTelemetry-based tracing

agent-skill-evals

Versioned evaluation, coverage, and release governance for Agent Skills

agent-steward

A quality gate and spend auditor for your AI agent fleet.

agent-suite-conformance

CLI contract v1 conformance kit for the agent suite (Plan 018 WI-2 / Plan 019 B1).

agent-tco

agent-tco: Pareto-optimal configuration of agentic workflows.

agent-tokenops

Run-aware token governance for agents: budgets, policies and halts on Chronicle and the AgentPlane control plane

agent-trajectory

Normalize native agent transcripts into a shared trajectory format.

agent-walkforward

Walk-forward / out-of-sample validation for agent evals — detect eval-set overfitting the way quants detect backtest overfitting.

agent-worklog

Renamed — this project continues as iiwi

agentacct

Local-first Agent Work Intelligence for coding agents: usage truth, recorded work, and honest joins

agentaudit-eval

Evaluation framework for multi-agent AI workflows: handoff quality scoring, failure attribution, loop detection, and cost guardrails.

agentcost-cli

Local CLI that profiles token spend for AI coding agent sessions (Claude, Cursor, Codex, Ollama).

agentgauge-harness

Statistical regression harness for MCP tool descriptions (measures whether a change actually moved agent task success) + a deterministic defect linter as a secondary utility

agentguard-evals

Local-first safety and reliability evaluation framework for AI coding agents.

agentguard-predict

Predict LLM-agent-run failures in real time — and intervene before the tokens are spent

agentharness-verifier

Independent, auditable verification for coding-agent claims

agentic-chaos

Standalone fault injection toolkit for LLM calls and agentic workflows, with an optional AgenticLens integration.

agentic-evalkit

Evaluate agentic systems with reproducible, evidence-first grading.

agenticmonitor

In-memory observability database for LLM agents: logs tool calls, messages, thinking, responses, tokens, and carried-in context.

agenticstack-costs

Token and cost tracking harness for AgenticStack: per-agent usage, USD estimates, and budgets

agentlatch

Zero-dependency, terminal-native Python middleware for resilient and observable AI agents.

agentlens-sdk-prakash

A powerful observability and debugging SDK for AI Agents

agentlint-trace

A CI linter for AI agent execution traces.

agentlog-ai

OpenTelemetry-native observability for multi-agent LLM systems

agentlog-tool

A local, offline digest of your coding-agent sessions. Zero dependencies.

agentloop-profiler

Profiler and optimization layer for AI agent execution graphs

agentloopguard-sdk

Detect and kill infinite AI agent loops before they burn your API budget.

agentmeter-cli

Monitor quota, tokens, and gateway balances for AI coding CLIs.

agentmeter-obs

Lightweight SDK to track AI agent tool calls, costs, and latency.

agentminutes

Parses agent harness session logs into a unified event schema. Python wrapper around the agentminutes Go CLI.

agento11y

Grafana Agent Observability Python SDK

agento11y-anthropic

Anthropic helper wrappers for the Grafana Agent Observability Python SDK

agento11y-claude-agent-sdk

Claude Agent SDK hooks for the Grafana Agent Observability Python SDK

agento11y-gemini

Gemini helper wrappers for the Grafana Agent Observability Python SDK

agento11y-google-adk

Google ADK callback handlers for the Grafana Agent Observability Python SDK

agento11y-langchain

LangChain callback handlers for the Grafana Agent Observability Python SDK

agento11y-langgraph

LangGraph callback handlers for the Grafana Agent Observability Python SDK

agento11y-litellm

LiteLLM callback handler for the Grafana Agent Observability Python SDK

agento11y-llamaindex

LlamaIndex callback handlers for the Grafana Agent Observability Python SDK

More in Agent frameworks & infrastructure