Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 57 of 57

lightpath-agent

Python agent for Lightpath: see which lines each HTTP request ran, live in your editor.

llama-index-evaluation-nace

LlamaIndex evaluators backed by Drex decision models from Nace

llama-index-selectors-nace

LlamaIndex selectors backed by Drex decision models from Nace

llm-app-ci-gate

One CI gate for LLM apps: prompt evaluation, security scan, and cost estimate in a single report.

ragcheck-ci

Regression checks for retrieval-augmented generation pipelines.

@forkpoint/agent-lighthouse-core

Core TypeScript engine for auditing websites for AI-agent readiness, LLM crawler access, MCP discovery, structured data, and agentic web signals

agent-regression-testing

A standalone library for AI agent regression testing using LLM-as-judge evaluation

@opafex/grounding

Deterministic checks that an LLM agent is backed by its tool results: fabricated identifiers, citation drift, self-contradiction, unsupported counts, unsupported money, percentages, counts, dates and reference numbers, invented written values, and - befor

@lmnr-ai/lmnr

TypeScript SDK for Laminar AI

@tensorcost/sdk

One-line wrapper SDK for TensorCost — fire-and-forget LLM observability for OpenAI, Anthropic, AWS Bedrock, and Google Vertex AI clients.

@gpambrozio/paseo-model-pricing

Paseo plugin: a sidebar table of what every model costs, across Anthropic, OpenAI, Fireworks AI, Ollama Cloud and OpenRouter

@trajekt-ai/core

Normalize AI agent traces (OpenTelemetry/OTLP JSON and protobuf, OpenInference, Langfuse, Vercel AI SDK), compute per-run metrics, detect regressions between versions, compare eval suites and assert on trajectories.

@spanport/sdk

Configure OpenTelemetry for Spanport, connect agent traces, and generate setup instructions for coding agents.

acdp-agent-hooks

Agent-framework hooks that sign tool results and final outputs and publish them to an ACDP registry, with derived_from lineage and OpenTelemetry attributes.

agent-tracedown

The flight recorder for AI agents: find the step that caused a failure, then replay the run with a fix.

agent-tracedown-server

Tracedown server and CLI: root-cause analysis for AI agent runs, importers for Langfuse and OpenTelemetry, replay planning and a web viewer.

agent-watch-otel

Agent Watch's OpenTelemetry span processor: production LLM spend as digests, never prompt text.

agentboxx

A tamper-evident flight recorder for AI agents

agenticml-lite

Lean, local-first experiment tracker for AgenticML. Installs as `agenticml-lite`, imports as `agenticml`.

agentstats

Evals for agent systems, with the statistics done right: CI gates, nightly checks, production monitoring from Langfuse/OpenTelemetry traces, and which models a smaller one could replace.

agenttrace-lite

Lightweight, local-first observability SDK for AI agents

agenturm

Terminal-first monitor for AI agent sessions, MCP calls, OpenTelemetry traces, and errors.

anosys-sdk-anthropic

AnoSys SDK for Anthropic - Automatic instrumentation and logging for Anthropic (Claude) API calls

anosys-sdk-claude-agents

AnoSys SDK for the Claude Agent SDK - Automatic tracing of Claude agents, tools and subagents

contragent

Contract-based deterministic trajectory supervision of LLM agents

langchain-tokenusage-dev

Read-only LangChain tools for tokenusage.dev Codex reset announcements and Nerf Watch scores.

llm-search-eval

Evaluate product search quality with IR metrics, multi-judge LLM labeling, cost tracking, and CI quality gates.

llm-spend-report

An llm CLI plugin that prices your logged usage: `llm spend` reports estimated cost by model or by day from logs.db

llmperfscope

Whole-lifecycle profiler for LLM inference engines: load, warmup, prefill, decode, serving and teardown; speed, memory and energy. Works with any engine, needs no code changes.

meterline-anthropic

Reads the Anthropic Enterprise Analytics API and emits meterline records.

meterline-anthropic-admin

Reads the Anthropic Admin Usage and Cost API of a Console organisation and emits meterline records.

meterline-litellm

Reads a LiteLLM proxy's admin API and emits meterline records.

pytest-llm-replay

Record LLM calls once, replay them in tests and CI: deterministic, free, with prompt-drift and cost reports.

tapedeck-llm

Flight recorder for LLM agents: record once, replay forever, diff what drifted. Zero dependencies.

traceAI-agent-framework

OpenTelemetry instrumentation for Microsoft Agent Framework

traceAI-agentscope

Future AGI adapter for AgentScope native OpenTelemetry tracing

traceAI-elevenlabs-agents

Forward ElevenLabs Agents post-call OpenTelemetry traces to Future AGI (Observe Voice tab)

traceAI-hermes-agent

Future AGI tracing plugin for Hermes Agent: turns, model calls and tool calls as OpenTelemetry spans

traceAI-openai-realtime

OpenTelemetry instrumentation for OpenAI Realtime WebSocket sessions (client.realtime.connect)

traceAI-semantic-kernel

Future AGI tracing for Semantic Kernel (Python) through its native OpenTelemetry diagnostics

Council of AI GSPC

Public measurement and verification agent for signed GSPC evidence across 22 axes. It exposes live A2A and MCP endpoints; measurement is not certification.

SuperBased

SuperBased is an open-source, local-first control plane for AI coding agents. One binary tracks provider-reported tokens and cost across Claude Code, Cursor, Codex, and ~40 tools, with dashboard terminals, live session takeover, model routing, and egress guardrails. Personal use free forever (Apache-2.0). Install: npx @superbased/observer

agent-lookout

Shows the AI agent sessions on your computer in one browser page, and which need you.

@sentrial/sdk

Sentrial SDK for TypeScript/Node.js - AI agent observability and monitoring

costcatch

Zero-instrumentation LLM agent tracer & cost tracker. Like `time` but for AI agents.

@webywz/agent-trace

Turn messy agent run logs into a readable timeline — filter, stats, Markdown & Mermaid export.

@camstack/addon-agent-ui

Agent UI — lightweight status dashboard served by every agent node

@javierfrancoleal/accountability-auditor

Agentic-workflow stage 3: mandatory reflection and verification pass that emits an Accountability Report before every final output.

@superpenguin/js

SuperPenguin JavaScript SDK for AI cost attribution across native provider SDKs, Vercel AI SDK, and OpenTelemetry

@xandreed/evals-langfuse

Efferent composable agent framework, 0.22.0. Node 24.

@xandreed/evals-langsmith

Efferent composable agent framework, 0.22.0. Node 24.

@xandreed/evals-cli

Efferent composable agent framework, 0.22.0. Node 24.

@javierfrancoleal/autonomy-grader

Governance layer of the J.Franco agentic workflow: grades the reliability of automatic decisions from the supervision log and proposes tier moves — the human approves every change.

agent-failure-lab

A catalogue of LLM agent failures with deterministic reproductions and tested guards.

agent-judge-framework

Agent-Judge-Framework: A Reliability-Oriented Framework for Automated Evaluation of Tool-Using Autonomous Agents

agentbill-io

OpenTelemetry-based SDK for tracking and billing AI agent usage with Cost Guard protection

agenteval-cx

Red-team and stress-test conversational AI agents with synthetic customers and safety probes — from your terminal, your CI, or the browser.

agentictracebook

Agentic Tracebook: review and retrieval layer for AI agent runs. Not released yet. A product of IO22 Labs.

langgraph-mini-dash

A visual debugging dashboard and UI for LangGraph multi-agent workflows

lbagents-eval

Evaluation of LbAgents workflows on replayed real failures

orkg-llm-logger

Log LLM calls (OpenAI, Gemini, OpenRouter, ...) to JSON/JSONL in the record format read by the @orkg/llm-logger dashboard.

rag-benchmark-kit

A lightweight toolkit for auditing and benchmarking RAG evaluation datasets.

AgentOps Apify Builders Bundle

Safer Apify agent ops for production builders (NimbliqueStudio / Zentra Foundry). Buyables: Bundle https://apify.com/zentrafoundry/agentops-apify-builders-bundle · Trace Normalizer https://apify.com/zentrafoundry/zentra-trace-normalizer · Tool Firewall https://apify.com/zentrafoundry/zentra-agentops-tool-firewall. Contact [email protected]

Evidence-bus starter

If your agents say "done" without a file path, you don't have proof — you have chat. $9 evidence-bus starter. Also $7 72h sprint checklist, $5 irreversible-action queue, $5 spend-receipt kit on Agensi.

AICost

AICost is an independent price desk for AI models, built around one question: what will this actually cost me? It tracks input, output and cache rates per million tokens for 300+ models from 30+ providers, including OpenAI, Anthropic, Google, Mistral, DeepSeek, Qwen and Kimi.

TokenTab

TokenTab estimates what an AI agent costs to run. Pick from 28 current LLMs (GPT, Claude, Gemini, Grok, DeepSeek, Mistral, Kimi and more), enter tokens per task, tool calls and caching, and compare cost per task and per month side by side. Free, no signup.

NovaSynth by Noveum

NovaSynth tests voice AI agents through synthetic callers with configurable personas, scenarios, interruptions and background noise. Phone and LiveKit endpoints; recordings, transcripts and evaluations. Proprietary hosted platform, account/setup required; freemium metered Noveum credits.

More in Agent frameworks & infrastructure