Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 48 of 57
LangGraph.js capture adapter for Provenance
Splyntra TypeScript SDK - Agent observability & security, built on OpenTelemetry
Local-first process assurance for agentic AI pipelines.
Open framework for simulating compatibility between agents — twin pairing, stress scenarios, no single match score
Release-readiness gates for AI agents: replay a pack of constructed safety scenarios, apply policy-as-code gates, and produce ship/warn/block evidence before an agent, prompt, model, or tool-policy change ships.
Local agentic benchmark suite for MLX and OpenAI-compatible inference engines.
CI/CD and Evaluation Infrastructure for Autonomous Agents (Agentic Engineering)
The Reliability Layer for Autonomous AI Agents — full tracing, cost intelligence, guardrails, and quality evaluation.
Free, open-source observability for AI agents. Trace LLM calls, catch silent failures, score output quality.
Agent Eye dashboard for monitoring GitHub Copilot CLI sessions in real-time
Profile, analyze, and optimize token consumption in LLM-powered applications and agentic workflows.
AgentLens Python SDK — zero-overhead observability for AI agents
Measure the real-world error rate and dollar cost of an AI agent's decisions. OpenTelemetry-native.
LLM Observability and Evaluation SDK
Make sense of what your AI agents did — local-first, MCP-native observability, trace replay & trajectory diff.
Deterministic snapshot testing harness for AI agents
Time-travel debugger and branch explorer for LangGraph AI agents. Capture execution traces, inspect LLM and tool calls, and branch from any point with overridden outputs.
System-level performance benchmark suite for agentic AI workloads (pre-release)
Open-source observability, cost tracking, and governance for AI agents
Quality gates and retry loops for Amazon Bedrock Converse.
Bench adapter for LangChain — auto-instrument every chain, agent, and LLM call.
Visualize AI agent step progress in the terminal
Windows tray app, overlay and dashboard for Claude plan limits (5-hour & weekly), with pace forecasts
Forge — AI Agent Harness SDK for Python. Wrap any AI agent with 12-dimension observation, 16 detectors, 14 autonomous intervention strategies.
Grade the generated response based on provided criteria.
IRIS compliance intelligence from LiteLLM config and proxy — keep your router, add governance
Standalone OpenTelemetry auto-instrumentation for jiuwenclaw / openjiuwen
LangChain integration for Tokonomics — AI cost metering, budget alerts, and spending caps for LLM API calls.
TealTiger governance callback handler for LlamaIndex — deterministic policy enforcement, cost tracking, and audit for RAG pipelines
Evaluate LLMs against behavioral specifications (AGENTS.md, Claude.md, custom rules)
Production-grade benchmarking platform for OpenAI-compatible LLM endpoints
Psychometric validation framework for LLM-generated classifications
LLM API latency benchmarker: TTFT p50/p95/p99 (cold/warm cache split), tokens/sec, cost.
Analyze and visualize how an LLM picks each next token: probabilities, attention, logit-lens, activations, manifold geometry — CLI + web app.
Interactive viewer for Copilot and Codex coding-session token usage
CLI-first, local LLM cost tracker for indie devs — drop it in front of any OpenAI-compatible API and watch token usage and spend in real time. No cloud, no signup.
LoongSuite Microsoft AutoGen instrumentation
LoongSuite DeepAgents Instrumentation
LoongSuite Microsoft Agent Framework instrumentation
Meterbility SDK for Python — trace LLM agent runs into the local Meterbility store
Multi-agent autonomous AI evaluation framework — simulate users, score dimensions, detect regressions
OpenBox SDK - Governance and observability for CrewAI agent workflows
OpenTelemetry LiteLLM instrumentation
Diagnose and repair why your LLM traces aren't showing up in PostHog.
pytest for RAG agents — behavioral audits with PASS/FAIL/WEAK verdicts
CLS observability SDK for OpenAI Agents SDK — 6-layer span hierarchy, direct upload to Tencent Cloud CLS
CLS observability SDK for vLLM — FastAPI middleware intercepts LLM requests, direct upload to Tencent Cloud CLS
RAG job pack for Tessera: compile a corpus + queries into a validated retrieval eval dataset.
CrewAI logging helpers for ZizkaDB — observability and causal debugging for multi-agent crews
LangChain callbacks for ZizkaDB — causal agent logging with parent_id lineage
Pluk structured events from AI agent terminal output — classify, subscribe, and react to JSONL event streams
Pydantic contracts and JSON Schemas for portable agent evaluation records.
Reusable proof profile, report, graph, and witness-planning primitives.
Makes a second request to the LLM, asking it if its original response was relevant to the prompt.
Publication-quality LLM architecture diagrams, fact sheets, and diffs from HuggingFace config.json — no weights needed
RAG observability: capture, evaluate, and explain pipeline runs (umbrella package + analyst CLI)
Pipeline instrumentation SDK for ragradar observability system
Shared schema, store, and target parsing for the ragradar observability system
Evaluation layer for ragradar observability system
Correlate AI coding agent token usage with git output to measure ROI
Report how much rate limit or subscription quota each registered Smithers account has consumed, across Claude, Codex, OpenAI, Anthropic, and Google providers.
OpenInference instrumentation for OpenAI Agents SDK
Enterprise tool for discovering AI Agents running within an organization
OpenTelemetry-based observability SDK for AI applications
Cost aggregation tool for Gas Town multi-agent workflows
agent-hotwash — analyze coding-agent traces to surface improvement opportunities and bad patterns
Official package name for the BenchCAD benchmark suite (placeholder release; the evaluation engine lives on GitHub)
Proxy Gemini (Vertex AI) completions wrapped in OpenTelemetry trace spans; returns the answer plus the trace/span id.
CLI for OpenAI and Anthropic organization cost reports
pytest for RAG systems - evaluate retrieval-augmented generation pipelines
Telemetry engine for DevFlow — record and analyze AI agent execution traces.
Raindrop integration for LangChain Deep Agents
Pi extension for monitoring Z.ai, DeepSeek, OpenAI Codex, and Opencode Go usage
LLM tracing SDK for auix — AI SDK middleware + manual tracing
Supervise external ACP AGENT runs with auditable, redacted local artifacts
Runtime control plane for AI agents: live cost metering and circuit breakers
OTel-native MeshAI connector for Claude Code: durable, evidence-grade telemetry for AI coding agent activity
A provider-agnostic toolkit for benchmarking retrieval strategies and running RAGAS end-to-end evaluation against your own PDF corpus.
Do your agent pipeline's own checks catch planted faults? Measured on LangGraph, CrewAI and AutoGen: median 16.7%. One prompt-level change takes it to 55.0%. Pre-registered spec, Apache-2.0 harness, every raw trace published.
Local observability for AI agent skill usage
Aeonic Node.js SDK for AI agent monitoring and drift detection
n8n AI Agent node with native Langfuse integration: tracing + prompt management
Overdeck Command Deck — AI agent monitoring desktop app
Pylva SDK — cost infrastructure for AI agent businesses
Non-blocking AI agent event logging for compliance teams
Detect which AI agent is running your code — structured identity (name, version, mode) with evidence, not just a boolean. Reads the AI_AGENT standard (both dialects) and the signals it misses.
Run wrud locally with one command - a local-first recorder for AI-agent sessions (tools, models, cost, insights). `npx @wrud/cli`.
Lightweight SDK to track AI agent tool calls, costs, and latency.
Deterministic behavioral audit harness for LLM agent transcripts — structural signature, rigidity trajectory, and sandbagging/eval-gaming detection. No model access, no LLM calls, reproducible.
LLM agent + LLM-as-Judge + Human-in-the-Loop review + learning loop, as an embeddable library
npm-installable CLI wrapper for agent-trace, deterministic record and replay for LLM agent HTTP traffic. Forwards straight to the real Python agent-trace CLI. Requires Python 3.10+ and the agent-observability-trace-cli PyPI package.
Static analysis CLI that finds deterministic LLM-agent cost-exposure bugs (unbounded retry loops, unbounded while(true) agents, leaked system prompts) and computes a worst-case dollar exposure ceiling from real, dated model pricing.
Opt-in OpenTelemetry (OTLP) trace + metrics publisher for the Agent Architecture Board. Streams LLM generations and container tool spans to any OTLP/HTTP backend — the workerd-safe fetch exporter on the Cloudflare Worker facade, the official @opentelemetr
Deterministic local eval utilities for AgentInspect traces
Rulvar evals: eval cases, golden outputs, rubric and judge graders, matrix sweeps, canary fingerprint.
Eval harness for Last Light workflows — drives the real production workflows against a mocked GitHub and grades deterministically (SWE-bench compatible).
A drop-in spend cap and audit log for LLM API calls
Rulvar executable store conformance kit (DEF-4).
Glassray Coach — a fully self-contained local AI-agent trace debugger (OTLP in, dashboard + agent-first CLI out).
Turn LangChain agent runs into portable Zansn evidence packs. Capture traces, tool calls, cost, and readiness with a callback handler.