Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 48 of 57

@peculis/provenance-langgraph

LangGraph.js capture adapter for Provenance

@splyntra/sdk

Splyntra TypeScript SDK - Agent observability & security, built on OpenTelemetry

agent-assure

Local-first process assurance for agentic AI pipelines.

agent-compat

Open framework for simulating compatibility between agents — twin pairing, stress scenarios, no single match score

agent-release-gates

Release-readiness gates for AI agents: replay a pack of constructed safety scenarios, apply policy-as-code gates, and produce ship/warn/block evidence before an agent, prompt, model, or tool-policy change ships.

agentblaster

Local agentic benchmark suite for MLX and OpenAI-compatible inference engines.

agentci-eval

CI/CD and Evaluation Infrastructure for Autonomous Agents (Agentic Engineering)

agentcontrolroom

The Reliability Layer for Autonomous AI Agents — full tracing, cost intelligence, guardrails, and quality evaluation.

agentdecode

Free, open-source observability for AI agents. Trace LLM calls, catch silent failures, score output quality.

agenteye-app

Agent Eye dashboard for monitoring GitHub Copilot CLI sessions in real-time

agenticlens

Profile, analyze, and optimize token consumption in LLM-powered applications and agentic workflows.

agentlens-py

AgentLens Python SDK — zero-overhead observability for AI agents

agentloss

Measure the real-world error rate and dollar cost of an AI agent's decisions. OpenTelemetry-native.

agentops-sdk

LLM Observability and Evaluation SDK

agentsense-ai

Make sense of what your AI agents did — local-first, MCP-native observability, trace replay & trajectory diff.

agentsnap

Deterministic snapshot testing harness for AI agents

agentstep

Time-travel debugger and branch explorer for LangGraph AI agents. Capture execution traces, inspect LLM and tool calls, and branch from any point with overridden outputs.

agentsysperf

System-level performance benchmark suite for agentic AI workloads (pre-release)

agentwatch-core

Open-source observability, cost tracking, and governance for AI agents

bedrockflow

Quality gates and retry loops for Amazon Bedrock Converse.

bench-langchain

Bench adapter for LangChain — auto-instrument every chain, agent, and LLM call.

claude-agentview

Visualize AI agent step progress in the terminal

claude-usage-tracker

Windows tray app, overlay and dashboard for Claude plan limits (5-hour & weekly), with pace forecasts

forge-agent-sdk

Forge — AI Agent Harness SDK for Python. Wrap any AI agent with 12-dimension observation, 16 detectors, 14 autonomous intervention strategies.

guardrails-ai-llm-critic

Grade the generated response based on provided criteria.

iris-litellm

IRIS compliance intelligence from LiteLLM config and proxy — keep your router, add governance

jiuwenswarm-instrumentor

Standalone OpenTelemetry auto-instrumentation for jiuwenclaw / openjiuwen

langchain-tokonomics

LangChain integration for Tokonomics — AI cost metering, budget alerts, and spending caps for LLM API calls.

llamaindex-tealtiger

TealTiger governance callback handler for LlamaIndex — deterministic policy enforcement, cost tracking, and audit for RAG pipelines

llm-behavioral-eval

Evaluate LLMs against behavioral specifications (AGENTS.md, Claude.md, custom rules)

llm-bench-studio

Production-grade benchmarking platform for OpenAI-compatible LLM endpoints

llm-classification-validator

Psychometric validation framework for LLM-generated classifications

llm-latency-bench

LLM API latency benchmarker: TTFT p50/p95/p99 (cold/warm cache split), tokens/sec, cost.

llm-token-heatmap

Analyze and visualize how an LLM picks each next token: probabilities, attention, logit-lens, activations, manifold geometry — CLI + web app.

llmly

Interactive viewer for Copilot and Codex coding-session token usage

llmstat

CLI-first, local LLM cost tracker for indie devs — drop it in front of any OpenAI-compatible API and watch token usage and spend in real time. No cloud, no signup.

loongsuite-instrumentation-autogen

LoongSuite Microsoft AutoGen instrumentation

loongsuite-instrumentation-deepagents

LoongSuite DeepAgents Instrumentation

loongsuite-instrumentation-microsoft-agent-framework

LoongSuite Microsoft Agent Framework instrumentation

meterbility-agent

Meterbility SDK for Python — trace LLM agent runs into the local Meterbility store

morpheus-tester-agents

Multi-agent autonomous AI evaluation framework — simulate users, score dimensions, detect regressions

openbox-crewai-sdk-python

OpenBox SDK - Governance and observability for CrewAI agent workflows

opentelemetry-instrumentation-litellm

OpenTelemetry LiteLLM instrumentation

posthog-llm-doctor

Diagnose and repair why your LLM traces aren't showing up in PostHog.

ragverdict

pytest for RAG agents — behavioral audits with PASS/FAIL/WEAK verdicts

tencentcloud-cls-sdk-openai

CLS observability SDK for OpenAI Agents SDK — 6-layer span hierarchy, direct upload to Tencent Cloud CLS

tencentcloud-cls-sdk-vllm

CLS observability SDK for vLLM — FastAPI middleware intercepts LLM requests, direct upload to Tencent Cloud CLS

tesserakit-rag

RAG job pack for Tessera: compile a corpus + queries into a validated retrieval eval dataset.

zizkadb-crewai

CrewAI logging helpers for ZizkaDB — observability and causal debugging for multi-agent crews

zizkadb-langchain

LangChain callbacks for ZizkaDB — causal agent logging with parent_id lineage

@kubestellar/pluk

Pluk structured events from AI agent terminal output — classify, subscribe, and react to JSONL event streams

agent-eval-contract

Pydantic contracts and JSON Schemas for portable agent evaluation records.

agentic-proofkit

Reusable proof profile, report, graph, and witness-planning primitives.

guardrails-ai-qa-relevance-llm-eval

Makes a second request to the LLM, asking it if its original response was relevant to the prompt.

llmviz

Publication-quality LLM architecture diagrams, fact sheets, and diffs from HuggingFace config.json — no weights needed

ragradar

RAG observability: capture, evaluate, and explain pipeline runs (umbrella package + analyst CLI)

ragradar-capture

Pipeline instrumentation SDK for ragradar observability system

ragradar-core

Shared schema, store, and target parsing for the ragradar observability system

ragradar-evaluate

Evaluation layer for ragradar observability system

codelens-ai

Correlate AI coding agent token usage with git output to measure ROI

@smithers-orchestrator/usage

Report how much rate limit or subscription quota each registered Smithers account has consumed, across Claude, Codex, OpenAI, Anthropic, and Google providers.

@arizeai/openinference-instrumentation-openai-agents

OpenInference instrumentation for OpenAI Agents SDK

ai-agent-discovery

Enterprise tool for discovering AI Agents running within an organization

@brizz/sdk

OpenTelemetry-based observability SDK for AI applications

gt-cost

Cost aggregation tool for Gas Town multi-agent workflows

agent-hotwash

agent-hotwash — analyze coding-agent traces to surface improvement opportunities and bad patterns

benchcad-agent

Official package name for the BenchCAD benchmark suite (placeholder release; the evaluation engine lives on GitHub)

gosce-traced-llm-mcp-proxy

Proxy Gemini (Vertex AI) completions wrapped in OpenTelemetry trace spans; returns the answer plus the trace/span id.

llm-org-cost-monitor

CLI for OpenAI and Anthropic organization cost reports

ragcheck-eval

pytest for RAG systems - evaluate retrieval-augmented generation pipelines

@devflow-tools/telemetry

Telemetry engine for DevFlow — record and analyze AI agent execution traces.

@raindrop-ai/deep-agents

Raindrop integration for LangChain Deep Agents

@beyona/pi-zai-usage

Pi extension for monitoring Z.ai, DeepSeek, OpenAI Codex, and Opencode Go usage

@aui-x/prism

LLM tracing SDK for auix — AI SDK middleware + manual tracing

agent-run-supervisor

Supervise external ACP AGENT runs with auditable, redacted local artifacts

agentfuse1

Runtime control plane for AI agents: live cost metering and circuit breakers

meshai-claude-code

OTel-native MeshAI connector for Claude Code: durable, evidence-grade telemetry for AI coding agent activity

raglens-toolkit

A provider-agnostic toolkit for benchmarking retrieval strategies and running RAGAS end-to-end evaluation against your own PDF corpus.

sabot

Do your agent pipeline's own checks catch planted faults? Measured on LangGraph, CrewAI and AutoGen: median 16.7%. One prompt-level change takes it to 55.0%. Pre-registered spec, Apache-2.0 harness, every raw trace published.

skilltrace

Local observability for AI agent skill usage

@aeonic-agentguard/sdk-node

Aeonic Node.js SDK for AI agent monitoring and drift detection

n8n-nodes-agent-langfuse

n8n AI Agent node with native Langfuse integration: tracing + prompt management

@overdeck/desktop

Overdeck Command Deck — AI agent monitoring desktop app

@pylva/sdk

Pylva SDK — cost infrastructure for AI agent businesses

sentinel-agent-sdk

Non-blocking AI agent event logging for compliance teams

ai-agent-detect

Detect which AI agent is running your code — structured identity (name, version, mode) with evidence, not just a boolean. Reads the AI_AGENT standard (both dialects) and the signals it misses.

@wrud/cli

Run wrud locally with one command - a local-first recorder for AI-agent sessions (tools, models, cost, insights). `npx @wrud/cli`.

@dnorio/agent-meter

Lightweight SDK to track AI agent tool calls, costs, and latency.

@af199/anima-eval

Deterministic behavioral audit harness for LLM agent transcripts — structural signature, rigidity trajectory, and sandbagging/eval-gaming detection. No model access, no LLM calls, reproducible.

hitl-agent

LLM agent + LLM-as-Judge + Human-in-the-Loop review + learning loop, as an embeddable library

agent-observability-trace-cli

npm-installable CLI wrapper for agent-trace, deterministic record and replay for LLM agent HTTP traffic. Forwards straight to the real Python agent-trace CLI. Requires Python 3.10+ and the agent-observability-trace-cli PyPI package.

valta-leak

Static analysis CLI that finds deterministic LLM-agent cost-exposure bugs (unbounded retry loops, unbounded while(true) agents, leaked system prompts) and computes a worst-case dollar exposure ceiling from real, dated model pricing.

@cat-factory/observability-otel

Opt-in OpenTelemetry (OTLP) trace + metrics publisher for the Agent Architecture Board. Streams LLM generations and container tool spans to any OTLP/HTTP backend — the workerd-safe fetch exporter on the Cloudflare Worker facade, the official @opentelemetr

@agent-inspect/eval

Deterministic local eval utilities for AgentInspect traces

@rulvar/evals

Rulvar evals: eval cases, golden outputs, rubric and judge graders, matrix sweeps, canary fingerprint.

lastlight-evals

Eval harness for Last Light workflows — drives the real production workflows against a mocked GitHub and grades deterministically (SWE-bench compatible).

@adarshsingh05/agent-guard

A drop-in spend cap and audit log for LLM API calls

@rulvar/store-conformance

Rulvar executable store conformance kit (DEF-4).

@glassray/coach

Glassray Coach — a fully self-contained local AI-agent trace debugger (OTLP in, dashboard + agent-first CLI out).

@zansn/langchain

Turn LangChain agent runs into portable Zansn evidence packs. Capture traces, tool calls, cost, and readiness with a callback handler.

More in Agent frameworks & infrastructure