Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 57 of 57
Python agent for Lightpath: see which lines each HTTP request ran, live in your editor.
LlamaIndex evaluators backed by Drex decision models from Nace
LlamaIndex selectors backed by Drex decision models from Nace
One CI gate for LLM apps: prompt evaluation, security scan, and cost estimate in a single report.
Regression checks for retrieval-augmented generation pipelines.
Core TypeScript engine for auditing websites for AI-agent readiness, LLM crawler access, MCP discovery, structured data, and agentic web signals
A standalone library for AI agent regression testing using LLM-as-judge evaluation
Deterministic checks that an LLM agent is backed by its tool results: fabricated identifiers, citation drift, self-contradiction, unsupported counts, unsupported money, percentages, counts, dates and reference numbers, invented written values, and - befor
TypeScript SDK for Laminar AI
One-line wrapper SDK for TensorCost — fire-and-forget LLM observability for OpenAI, Anthropic, AWS Bedrock, and Google Vertex AI clients.
Paseo plugin: a sidebar table of what every model costs, across Anthropic, OpenAI, Fireworks AI, Ollama Cloud and OpenRouter
Normalize AI agent traces (OpenTelemetry/OTLP JSON and protobuf, OpenInference, Langfuse, Vercel AI SDK), compute per-run metrics, detect regressions between versions, compare eval suites and assert on trajectories.
Configure OpenTelemetry for Spanport, connect agent traces, and generate setup instructions for coding agents.
Agent-framework hooks that sign tool results and final outputs and publish them to an ACDP registry, with derived_from lineage and OpenTelemetry attributes.
The flight recorder for AI agents: find the step that caused a failure, then replay the run with a fix.
Tracedown server and CLI: root-cause analysis for AI agent runs, importers for Langfuse and OpenTelemetry, replay planning and a web viewer.
Agent Watch's OpenTelemetry span processor: production LLM spend as digests, never prompt text.
A tamper-evident flight recorder for AI agents
Lean, local-first experiment tracker for AgenticML. Installs as `agenticml-lite`, imports as `agenticml`.
Evals for agent systems, with the statistics done right: CI gates, nightly checks, production monitoring from Langfuse/OpenTelemetry traces, and which models a smaller one could replace.
Lightweight, local-first observability SDK for AI agents
Terminal-first monitor for AI agent sessions, MCP calls, OpenTelemetry traces, and errors.
AnoSys SDK for Anthropic - Automatic instrumentation and logging for Anthropic (Claude) API calls
AnoSys SDK for the Claude Agent SDK - Automatic tracing of Claude agents, tools and subagents
Contract-based deterministic trajectory supervision of LLM agents
Read-only LangChain tools for tokenusage.dev Codex reset announcements and Nerf Watch scores.
Evaluate product search quality with IR metrics, multi-judge LLM labeling, cost tracking, and CI quality gates.
An llm CLI plugin that prices your logged usage: `llm spend` reports estimated cost by model or by day from logs.db
Whole-lifecycle profiler for LLM inference engines: load, warmup, prefill, decode, serving and teardown; speed, memory and energy. Works with any engine, needs no code changes.
Reads the Anthropic Enterprise Analytics API and emits meterline records.
Reads the Anthropic Admin Usage and Cost API of a Console organisation and emits meterline records.
Reads a LiteLLM proxy's admin API and emits meterline records.
Record LLM calls once, replay them in tests and CI: deterministic, free, with prompt-drift and cost reports.
Flight recorder for LLM agents: record once, replay forever, diff what drifted. Zero dependencies.
OpenTelemetry instrumentation for Microsoft Agent Framework
Future AGI adapter for AgentScope native OpenTelemetry tracing
Forward ElevenLabs Agents post-call OpenTelemetry traces to Future AGI (Observe Voice tab)
Future AGI tracing plugin for Hermes Agent: turns, model calls and tool calls as OpenTelemetry spans
OpenTelemetry instrumentation for OpenAI Realtime WebSocket sessions (client.realtime.connect)
Future AGI tracing for Semantic Kernel (Python) through its native OpenTelemetry diagnostics
Public measurement and verification agent for signed GSPC evidence across 22 axes. It exposes live A2A and MCP endpoints; measurement is not certification.
SuperBased is an open-source, local-first control plane for AI coding agents. One binary tracks provider-reported tokens and cost across Claude Code, Cursor, Codex, and ~40 tools, with dashboard terminals, live session takeover, model routing, and egress guardrails. Personal use free forever (Apache-2.0). Install: npx @superbased/observer
Shows the AI agent sessions on your computer in one browser page, and which need you.
Sentrial SDK for TypeScript/Node.js - AI agent observability and monitoring
Zero-instrumentation LLM agent tracer & cost tracker. Like `time` but for AI agents.
Turn messy agent run logs into a readable timeline — filter, stats, Markdown & Mermaid export.
Agent UI — lightweight status dashboard served by every agent node
Agentic-workflow stage 3: mandatory reflection and verification pass that emits an Accountability Report before every final output.
SuperPenguin JavaScript SDK for AI cost attribution across native provider SDKs, Vercel AI SDK, and OpenTelemetry
Efferent composable agent framework, 0.22.0. Node 24.
Efferent composable agent framework, 0.22.0. Node 24.
Efferent composable agent framework, 0.22.0. Node 24.
Governance layer of the J.Franco agentic workflow: grades the reliability of automatic decisions from the supervision log and proposes tier moves — the human approves every change.
A catalogue of LLM agent failures with deterministic reproductions and tested guards.
Agent-Judge-Framework: A Reliability-Oriented Framework for Automated Evaluation of Tool-Using Autonomous Agents
OpenTelemetry-based SDK for tracking and billing AI agent usage with Cost Guard protection
Red-team and stress-test conversational AI agents with synthetic customers and safety probes — from your terminal, your CI, or the browser.
Agentic Tracebook: review and retrieval layer for AI agent runs. Not released yet. A product of IO22 Labs.
A visual debugging dashboard and UI for LangGraph multi-agent workflows
Evaluation of LbAgents workflows on replayed real failures
Log LLM calls (OpenAI, Gemini, OpenRouter, ...) to JSON/JSONL in the record format read by the @orkg/llm-logger dashboard.
A lightweight toolkit for auditing and benchmarking RAG evaluation datasets.
Safer Apify agent ops for production builders (NimbliqueStudio / Zentra Foundry). Buyables: Bundle https://apify.com/zentrafoundry/agentops-apify-builders-bundle · Trace Normalizer https://apify.com/zentrafoundry/zentra-trace-normalizer · Tool Firewall https://apify.com/zentrafoundry/zentra-agentops-tool-firewall. Contact [email protected]
If your agents say "done" without a file path, you don't have proof — you have chat. $9 evidence-bus starter. Also $7 72h sprint checklist, $5 irreversible-action queue, $5 spend-receipt kit on Agensi.
AICost is an independent price desk for AI models, built around one question: what will this actually cost me? It tracks input, output and cache rates per million tokens for 300+ models from 30+ providers, including OpenAI, Anthropic, Google, Mistral, DeepSeek, Qwen and Kimi.
TokenTab estimates what an AI agent costs to run. Pick from 28 current LLMs (GPT, Claude, Gemini, Grok, DeepSeek, Mistral, Kimi and more), enter tokens per task, tool calls and caching, and compare cost per task and per month side by side. Free, no signup.
NovaSynth tests voice AI agents through synthetic callers with configurable personas, scenarios, interruptions and background noise. Phone and LiveKit endpoints; recordings, transcripts and evaluations. Proprietary hosted platform, account/setup required; freemium metered Noveum credits.