Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 56 of 57
Limit AI usage by subscription plan and track AI cost per user against revenue (RevenueCat, Stripe or any billing). Zero-dependency SDK for your backend.
CoreWeave Forge Agent Lens tracing for Google Antigravity
Benchmark latency, throughput, cost and quantisation quality loss of any OpenAI-compatible LLM endpoint.
Local-first receipts and safe undo for AI agents
Evals with multi-agent swarms (experimental).
Kyvvu listener for CrewAI
Kyvvu callback handlers for LangChain and LangGraph
Per-request serving-intent hints (X-Agent-Hint) and out-of-band agent status events, as LangChain agent middleware.
LangChain and LangGraph callback for AI usage costs, customer attribution and Veqen budgets.
Graph of Agentic Thoughts: Bridging Agentic Reasoning and Explainability through Graphs
ExtractBench: a benchmark and evaluation harness for schema-guided document extraction on real-world enterprise documents
Llama Stack out-of-tree eval provider (remote::csoai): verify CSOAI signed evidence batches offline and return each event's state, never a score.
Evaluate and observe LLM-powered applications with deterministic metrics and LLM-as-Judge.
Record and replay LLM agent SDK and tool calls for offline pytest.
A local CLI tool for evaluating LLM prompt outputs against test suites
CI/CD quality gate framework for LLM and RAG applications: metric, schema, and security testing with a pytest-native workflow.
Catch silent LLM failures: truncation, reasoning-budget exhaustion, empty output, content filters.
Turn any LLM into a decision model: a probability for each predefined option of a question about a text, through a Jev-compatible API
Lightweight Python caching, resilience, semantic caching, rate limiting and observability for LLM/API applications.
Benchmark Report: a standard data format for LLM inference benchmark runs (schemas, validators, and converters)
A debugger for stepping through LLM layers and interpreting internal representations.
Python (pytest) client for llmentary: deterministic trace-based integration tests for LLM apps
LLM FLock is a CLI tool to interact with local and remote LLM, define multiple scenario variations (model × prompt × options), run them over a dataset and capture structured results for comparison or to build knowledge through meta analysis.
Reports, quick functions and live training tracking for any PyTorch language model: architecture, health, speed, perplexity, behavior, code, and a live training dashboard with alerts.
Switch AI models without breaking your product: find model calls, turn real usage into tests, compare models, fix prompts and watch for drift.
Live OpenRouter LLM prices + Artificial Analysis intelligence & coding benchmarks dashboard
Signed, tamper-evident decision evidence for CrewAI Agents using LoopGrid
Signed, tamper-evident decision evidence for Haystack Agents using LoopGrid
Native LangChain agent middleware for LoopGrid signed decision evidence, policy gates and guarded tool execution
Native LangGraph decision evidence, application policy, human-review gates and signed verification for LoopGrid
Native decision evidence middleware for Microsoft Agent Framework and LoopGrid
Signed, tamper-evident decision evidence for OpenAI Agents SDK workflows using LoopGrid
Instrument your own AI agents (LangChain, LangGraph, CrewAI, OpenAI Agents SDK, Google ADK, or plain Python) for Metano: session telemetry and runtime guardrails.
Telemetry dependencies for the Mistral Apps Observability capability.
MA-Trace: causal tracing, coordination SLOs and deterministic replay for multi-agent LLM systems (multi-agent observability on OpenTelemetry).
Measure an LLM judge's retest variance before you trust its deltas.
Shared library for LLM calls at MIT Open Learning, with Opik tracing, prompt management, and evaluation built in
Anthropic infrastructure FinOps SDK — scan, analyze, and export your Anthropic org resources
OpenAI infrastructure FinOps SDK — scan, analyze, and export your OpenAI org resources
py3status module showing Claude Code subscription usage
Evidence-based debugging and root-cause analysis for AI agents and LLM applications
Evidence-bound evaluation, verification, validation, and release assurance for agentic systems
Lightweight tool execution tracing SDK for evaluating AI agents on QAgent.
Lightweight adversarial hallucination auditor and citation grounding engine for LangGraph and production RAG pipelines.
RAGAS with a memory and a diff view. Evaluate RAG pipelines, persist runs, and fail CI on regressions.
A lightweight Python wrapper for evaluating RAG pipelines using Ragas.
A local failure debugger for RAG applications: isolates retrieval, ranking, generation, and abstention failures.
ModernBERT-based RAG evaluation model
Administrative and read API clients for Redis Lens
Apples-to-apples LLM cost & performance benchmark — cost per successful task, reproducible.
AER-1 verifiable execution receipts for smolagents agents: one checkable receipt per tool call.
Hard budget caps for subagents — token/$ limits enforced by hooks, not just displayed
Team/CI budget control for subagent token spend: shared budget pools, per-agent quotas, overspend checks, and a kill switch.
Estimate how much of your Claude Code bill is subagent preamble resends
Evidence-based operations for AI agent fleets: sealed audit trails, SLA-alarmed triage and AI-Act-ready privacy.
Universal Multi-Agent Observability, Entropy Circuit-Breaker, and Time-Machine Debugging Platform
The SDK for agents on the Synadia Agent Fabric — tracing and the agent tools' ScratchPad extension, on the hooks of the Synadia Agent Protocol SDK
SZL Evidence Plane plugin for LiteLLM: hash-chained, DSSE-signed receipts for every LLM request via LiteLLM's callback system.
Records what an AI agent spent, from inside the agent's own process, for Agent Pulse.
The Tellproof evidence agent: a signed, content-free record of how your AI system runs, with a local dashboard.
The Tenbri Agent Decision Record for the Claude Agent SDK: every tool call a signed, provable decision.
The Tenbri Agent Decision Record for LangGraph and LangChain: every tool call a signed, chained decision.
The Tenbri Agent Decision Record for the OpenAI Agents SDK: every tool call, handoff and approval signed, chained and provable.
Retrieval regression testing in CI: trec_eval-equal metrics, per-query diffs, significance tests and leakage checks.
The Tracepanel agent: keeps your app's traces and logs, finds N+1 queries and other issues, serves the local console
Ship the errors your Python app already logs to UptimeAgent, so incidents arrive with your own error lines attached. Evidence only, never a vote on uptime. Zero dependencies.
A live map of which agent in a multi-agent AI system is doing what - from its chain of thought.
Run script-defined vLLM benchmarks through Docker Compose or Modal.
Eval harness for Claude Code agents: golden tasks, rule checks, LLM-judge rubrics, traces, cost/latency, regression gate.
LangGraph middleware for ZizkaDB — automatic causal logging per graph node
AI Humanizer Benchmark review: compare features, pricing, use cases, access model, and alternatives for this AI Detection agent in 2026. Independent monthly benchmarks of AI humanizers across detection performance and writing quality.
Lightweight self-hosted Agent Operations (AgentOps / LLMOps) toolkit
Deterministic critic for AI agents: runs weighted checks for real, scores 0-100 and blocks the delivery when it should.
Report Claude Code, Cursor, Gemini CLI, OpenCode and Cortex Code sessions to the AgentLedger Snowflake Native App.
A built-in acceptance test suite for self-hosted LLM services
Multi-framework agent retrieval tracing (LangGraph, LlamaIndex, CrewAI) rendered as interactive graphs with graphsight-pro.
Ask your coding agent about your LLM app in plain English; it answers from your Langfuse data with lf. CLI + agent skill.
Multi-Router monitoring dashboard with automatic detection and SGLang worker maintenance
Provider-neutral effect assurance and reliability testing for agent tool operations.
Automatically trace OpenCode conversations to Braintrust. Captures user messages, assistant responses, and tool calls for observability.
In-process mock of LLM provider APIs (OpenAI, Anthropic, Gemini, Bedrock) for testing apps with the official SDKs — no tokens spent.
Universal telemetry and digital receipt generator for the AI coding agent era
Audit trail database for AI agents: tamper-evident, checksum-backed decision logs, session replay and time-travel debugging to support EU AI Act Article 12 record-keeping.
OpenTelemetry tracing and Prometheus metrics for BanglaClaw
Agent evaluation comparability semantics: spec + conformance suite + thin reference implementation
Python client for Agent Referee (agentreferee.ai): sealed review of an AI agent's output by independent AI reviewers, fair tie-breaks and AI judges between agents, and offline checks of the signed receipts.
Find where your AI coding agent wastes tokens, and how to stop it
Pydantic AI capability integration for TraceAgent.
Standalone HTML, Markdown, and view-model reporting engine for Agent Lighthouse AI-agent readiness audits
Price a whole LLM agent run, not one request. Models context regrowth across turns, prompt-cache write premiums and expiry, the minimum cacheable prefix, long-context rates, tool-definition overhead and thinking tokens.
AgentLens callback handler for LangChain and LangGraph: traces and feedback
Anthropic and OpenAI Codex quota status extension for pi
Hue OpenTelemetry helpers for Node.js agent applications
Tenbri for LangGraph and LangChain.js: every tool call a signed, chained decision.
Structural verification of multiagent systems
Chaos engineering for AI agents: break tools, models and context on purpose, then measure whether your agent survives.
Reads the Claude Code logs already on your machine and tells you which agent's 'done' you can't trust, and what to run next.
Provider-agnostic evaluation and benchmarking platform for autonomous AI coding agents
ChatSee connector for the Anthropic Messages API — capture every turn with one line.
ChatSee connector for the Claude Agent SDK — capture every turn with one line.