Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 56 of 57

fiscallm

Limit AI usage by subscription plan and track AI cost per user against revenue (RevenueCat, Stripe or any billing). Zero-dependency SDK for your backend.

forge-agent-lens-for-google-antigravity

CoreWeave Forge Agent Lens tracing for Google Antigravity

frugal-llm

Benchmark latency, throughput, cost and quantisation quality loss of any OpenAI-compatible LLM endpoint.

glassbox-agent

Local-first receipts and safe undo for AI agents

inspect-swarm

Evals with multi-agent swarms (experimental).

kyvvu-crewai

Kyvvu listener for CrewAI

kyvvu-langchain

Kyvvu callback handlers for LangChain and LangGraph

langchain-agent-hints

Per-request serving-intent hints (X-Agent-Hint) and out-of-band agent status events, as LangChain agent middleware.

langchain-veqen

LangChain and LangGraph callback for AI usage costs, customer attribution and Veqen budgets.

langgraph-goat

Graph of Agentic Thoughts: Bridging Agentic Reasoning and Explainability through Graphs

llama-extract-bench

ExtractBench: a benchmark and evaluation harness for schema-guided document extraction on real-world enterprise documents

llama-stack-provider-csoai

Llama Stack out-of-tree eval provider (remote::csoai): verify CSOAI signed evidence batches offline and return each event's state, never a score.

llm-assayer-eval

Evaluate and observe LLM-powered applications with deterministic metrics and LLM-as-Judge.

llm-cassette

Record and replay LLM agent SDK and tool calls for offline pytest.

llm-eval-bench

A local CLI tool for evaluating LLM prompt outputs against test suites

llm-eval-guard

CI/CD quality gate framework for LLM and RAG applications: metric, schema, and security testing with a pytest-native workflow.

llm-stopcheck

Catch silent LLM failures: truncation, reasoning-budget exhaustion, empty output, content filters.

llm2decision

Turn any LLM into a decision model: a probability for each predefined option of a question about a text, through a Jev-compatible API

llmcachex-core

Lightweight Python caching, resilience, semantic caching, rate limiting and observability for LLM/API applications.

llmd-benchmark-report

Benchmark Report: a standard data format for LLM inference benchmark runs (schemas, validators, and converters)

llmdbg

A debugger for stepping through LLM layers and interpreting internal representations.

llmentary

Python (pytest) client for llmentary: deterministic trace-based integration tests for LLM apps

llmflock

LLM FLock is a CLI tool to interact with local and remote LLM, define multiple scenario variations (model × prompt × options), run them over a dataset and capture structured results for comparison or to build knowledge through meta analysis.

llmreport

Reports, quick functions and live training tracking for any PyTorch language model: architecture, health, speed, perplexity, behavior, code, and a live training dashboard with alerts.

llmshift

Switch AI models without breaking your product: find model calls, turn real usage into tests, compare models, fix prompts and watch for drift.

llmval

Live OpenRouter LLM prices + Artificial Analysis intelligence & coding benchmarks dashboard

loopgrid-crewai

Signed, tamper-evident decision evidence for CrewAI Agents using LoopGrid

loopgrid-haystack

Signed, tamper-evident decision evidence for Haystack Agents using LoopGrid

loopgrid-langchain

Native LangChain agent middleware for LoopGrid signed decision evidence, policy gates and guarded tool execution

loopgrid-langgraph

Native LangGraph decision evidence, application policy, human-review gates and signed verification for LoopGrid

loopgrid-microsoft-agent

Native decision evidence middleware for Microsoft Agent Framework and LoopGrid

loopgrid-openai-agents

Signed, tamper-evident decision evidence for OpenAI Agents SDK workflows using LoopGrid

metano-agent-sdk

Instrument your own AI agents (LangChain, LangGraph, CrewAI, OpenAI Agents SDK, Google ADK, or plain Python) for Metano: session telemetry and runtime guardrails.

mistralai-capabilities-feature-observability

Telemetry dependencies for the Mistral Apps Observability capability.

multi-agent-observability

MA-Trace: causal tracing, coordination SLOs and deterministic replay for multi-agent LLM systems (multi-agent observability on OpenTelemetry).

noisefloor-llm

Measure an LLM judge's retest variance before you trust its deltas.

ol-llm

Shared library for LLM calls at MIT Open Learning, with Opik tracing, prompt management, and evaluation built in

pump-anthropic-radar

Anthropic infrastructure FinOps SDK — scan, analyze, and export your Anthropic org resources

pump-openai-radar

OpenAI infrastructure FinOps SDK — scan, analyze, and export your OpenAI org resources

py3status-claude-usage

py3status module showing Claude Code subscription usage

pyagenthound

Evidence-based debugging and root-cause analysis for AI agents and LLM applications

qa-automation-ai-agent-evals

Evidence-bound evaluation, verification, validation, and release assurance for agentic systems

qagent-tracer

Lightweight tool execution tracing SDK for evaluating AI agents on QAgent.

rag-critic

Lightweight adversarial hallucination auditor and citation grounding engine for LangGraph and production RAG pipelines.

rag-eval-harness

RAGAS with a memory and a diff view. Evaluate RAG pipelines, persist runs, and fail CI on regressions.

rag-eval-kit-avan

A lightweight Python wrapper for evaluating RAG pipelines using Ragas.

ragmortem

A local failure debugger for RAG applications: isolates retrieval, ranking, generation, and abstention failures.

ragone-ai

ModernBERT-based RAG evaluation model

redis-agent-lens

Administrative and read API clients for Redis Lens

sf-smartllmcost

Apples-to-apples LLM cost & performance benchmark — cost per successful task, reproducible.

smolagents-aer1

AER-1 verifiable execution receipts for smolagents agents: one checkable receipt per tool call.

subagent-budget

Hard budget caps for subagents — token/$ limits enforced by hooks, not just displayed

subagent-budget-team

Team/CI budget control for subagent token spend: shared budget pools, per-agent quotas, overspend checks, and a kill switch.

subagent-tax

Estimate how much of your Claude Code bill is subagent preamble resends

swarmax-fleet

Evidence-based operations for AI agent fleets: sealed audit trails, SLA-alarmed triage and AI-Act-ready privacy.

swarmwatch

Universal Multi-Agent Observability, Entropy Circuit-Breaker, and Time-Machine Debugging Platform

synadia-ai-agent-fabric

The SDK for agents on the Synadia Agent Fabric — tracing and the agent tools' ScratchPad extension, on the hooks of the Synadia Agent Protocol SDK

szl-evidence-litellm

SZL Evidence Plane plugin for LiteLLM: hash-chained, DSSE-signed receipts for every LLM request via LiteLLM's callback system.

techbridge-agentpulse

Records what an AI agent spent, from inside the agent's own process, for Agent Pulse.

tellproof-agent

The Tellproof evidence agent: a signed, content-free record of how your AI system runs, with a local dashboard.

tenbri-claude-agent-sdk

The Tenbri Agent Decision Record for the Claude Agent SDK: every tool call a signed, provable decision.

tenbri-langgraph

The Tenbri Agent Decision Record for LangGraph and LangChain: every tool call a signed, chained decision.

tenbri-openai-agents

The Tenbri Agent Decision Record for the OpenAI Agents SDK: every tool call, handoff and approval signed, chained and provable.

toolkit-rag-quality

Retrieval regression testing in CI: trec_eval-equal metrics, per-query diffs, significance tests and leakage checks.

tracepanel-agent

The Tracepanel agent: keeps your app's traces and logs, finds N+1 queries and other issues, serves the local console

uptime-agent

Ship the errors your Python app already logs to UptimeAgent, so incidents arrive with your own error lines attached. Evidence only, never a vote on uptime. Zero dependencies.

visualagents

A live map of which agent in a multi-agent AI system is doing what - from its chain of thought.

vllm-benchmark

Run script-defined vLLM benchmarks through Docker Compose or Modal.

wm-agent-evals

Eval harness for Claude Code agents: golden tasks, rule checks, LLM-judge rubrics, traces, cost/latency, regression gate.

zizkadb-langgraph

LangGraph middleware for ZizkaDB — automatic causal logging per graph node

ai-humanizer-benchmark

AI Humanizer Benchmark review: compare features, pricing, use cases, access model, and alternatives for this AI Detection agent in 2026. Independent monthly benchmarks of AI humanizers across detection performance and writing quality.

agent-operations

Lightweight self-hosted Agent Operations (AgentOps / LLMOps) toolkit

agent-test-loop

Deterministic critic for AI agents: runs weighted checks for real, scores 0-100 and blocks the delivery when it should.

agentledger-cli

Report Claude Code, Cursor, Gemini CLI, OpenCode and Cortex Code sessions to the AgentLedger Snowflake Native App.

giraffe-llm

A built-in acceptance test suite for self-hosted LLM services

graphsight-pro-langgraph

Multi-framework agent retrieval tracing (LangGraph, LlamaIndex, CrewAI) rendered as interactive graphs with graphsight-pro.

langfuse-agent-cli

Ask your coding agent about your LLM app in plain English; it answers from your Langfuse data with lf. CLI + agent skill.

llm-router-monitor

Multi-Router monitoring dashboard with automatic detection and SGLang worker maintenance

production-agent-systems

Provider-neutral effect assurance and reliability testing for agent tool operations.

@braintrust/trace-opencode

Automatically trace OpenCode conversations to Braintrust. Captures user messages, assistant responses, and tool calls for observability.

mock-llm

In-process mock of LLM provider APIs (OpenAI, Anthropic, Gemini, Bedrock) for testing apps with the official SDKs — no tokens spent.

qodewk

Universal telemetry and digital receipt generator for the AI coding agent era

zizkadb-sdk

Audit trail database for AI agents: tamper-evident, checksum-backed decision logs, session replay and time-travel debugging to support EU AI Act Article 12 record-keeping.

@entrogic-net/observability

OpenTelemetry tracing and Prometheus metrics for BanglaClaw

agent-comparability

Agent evaluation comparability semantics: spec + conformance suite + thin reference implementation

agent-referee-client

Python client for Agent Referee (agentreferee.ai): sealed review of an AI agent's output by independent AI reviewers, fair tie-breaks and AI judges between agents, and offline checks of the signed receipts.

agentwaste

Find where your AI coding agent wastes tokens, and how to stop it

trace-agent-pydantic-ai

Pydantic AI capability integration for TraceAgent.

@forkpoint/agent-lighthouse-report

Standalone HTML, Markdown, and view-model reporting engine for Agent Lighthouse AI-agent readiness audits

@antharmaya/tokencost

Price a whole LLM agent run, not one request. Models context regrowth across turns, prompt-cache write premiums and expiry, the minimum cacheable prefix, long-context rates, tool-definition overhead and thinking tokens.

@linnify/agentlens-langchain

AgentLens callback handler for LangChain and LangGraph: traces and feedback

@ravshansbox/pi-quota

Anthropic and OpenAI Codex quota status extension for pi

@hue-run/sdk

Hue OpenTelemetry helpers for Node.js agent applications

@tenbri/langgraph

Tenbri for LangGraph and LangChain.js: every tool call a signed, chained decision.

agent-certify

Structural verification of multiagent systems

agent-chaos-toolkit

Chaos engineering for AI agents: break tools, models and context on purpose, then measure whether your agent survives.

agent-postmortem

Reads the Claude Code logs already on your machine and tells you which agent's 'done' you can't trust, and what to run next.

agentbench-core

Provider-agnostic evaluation and benchmarking platform for autonomous AI coding agents

chatsee-anthropic

ChatSee connector for the Anthropic Messages API — capture every turn with one line.

chatsee-claude-agent-sdk

ChatSee connector for the Claude Agent SDK — capture every turn with one line.

More in Agent frameworks & infrastructure