Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 44 of 57

trulens-apps-langgraph

Library to systematically track and evaluate LangGraph based applications.

trulens-apps-llamaindex

Library to systematically track and evaluate LLM based applications.

trulens-instrument-langchain

Library to systematically track and evaluate LLM based applications.

trulens-instrument-llamaindex

Library to systematically track and evaluate LLM based applications.

trulens-providers-bedrock

Library to systematically track and evaluate LLM based applications.

trulens-providers-langchain

Library to systematically track and evaluate LLM based applications.

trulens-providers-litellm

Library to systematically track and evaluate LLM based applications.

trulens-providers-openai

Library to systematically track and evaluate LLM based applications.

trustllm

TrustLLM

trustrag-eval

RAGAS-based evaluation pipeline with trust-specific metrics for TrustRAG

truthscore-llm

Claim-grounded truth evaluation for LLM outputs: retrieval, per-claim verification, and explicit risk signals

tui-agent-dashboard

Beautiful terminal UI for monitoring AI agent operations, costs, and attribution

tukey-llm

Compare LLM responses side-by-side with local persistence

turingpulse-sdk-anthropic

TuringPulse SDK integration for Anthropic

turingpulse-sdk-bedrock

TuringPulse SDK integration for AWS Bedrock

turingpulse-sdk-cohere

TuringPulse SDK integration for Cohere

turingpulse-sdk-mistral

TuringPulse SDK integration for Mistral AI

turingpulse-sdk-openai

TuringPulse SDK integration for OpenAI

tutorbot-safety-agent

Safety Agent for evaluating TutorBot AI responses

twelve-angry-llms

Panel-based reliability annotation for preference data: measure inter-judge agreement (IJA) with a panel of LLM judges and export TRL-ready DPO data with reliability signals attached.

umbrage-evals

Umbrage Evals SDK

uptrain-haystack

An integration of UpTrain LLM evaluation framework with Haystack

useagentlens

See where your LLM money goes. Get told exactly how to cut it.

validated-llm

LLM output validation with retry loops and optional Reflexion

valohai-llm

Track LLM evaluations to Valohai LLM

vantage-agent

Vantage Agent

vcr-langchain

Record and replay LLM interactions for langchain

vectry-agentlens

Agent observability that traces decisions, not just API calls

verifai-llm-sdk

VerifAI Python SDK for LLM tracing and evaluation

veritasagent-sdk

Tamper-proof audit telemetry for AI agents

viatoris-langchain

LangChain callback handler for Viatoris — emit a signed Viatoris receipt for every LangChain tool call

visuallm

Visualization tool for various generation tasks on Language Models.

vital-agent-eval-env

Vital Agent Eval Env

vllm-benchmark-suite

The most comprehensive benchmarking suite for vLLM inference servers

vllm-doctor

Diagnostic tool for vLLM inference servers

vllm-htop

htop-style terminal monitor for vLLM inference servers

vllm-judge

LLM-as-a-Judge evaluations for vLLM hosted models

vllm-lens

vLLM plugin for interacting with activations during inference

vllm-mon

Production-grade vLLM metrics monitoring TUI with persistent storage and Grafana-style visualizations

vllm-semantic-router-bench

Comprehensive benchmark suite for semantic router vs direct vLLM evaluation across multiple reasoning datasets

vllm-sr-sim

vLLM Semantic Router fleet simulator for capacity planning, SLO validation, and what-if analysis

vllm-top

A monitoring tool for vLLM metrics.

vllmoni

A CLI tool for monitoring and managing VLLM inference servers in Docker containers

voice-of-agents

Synthetic user research and LLM eval harness — research-grade rigor for developers who can't afford a research team.

voltagent

Python SDK for VoltAgent - AI Agent Observability and Tracing

volttron-agent-watcher

Agent Watcher is used to monitor agents running on a VOLTTRON instance

voxagent-bench

coming soon

voxagent-gym

coming soon

wandb-agentstream

Read, transform, and write AI agent session logs across formats

watchagent

Debugger and monitor for AI agents

watchllm

Unified CLI for the WatchLLM governance ecosystem.

weave-haystack

An integration of Weights & Biases Weave tracer with Haystack

weights-bias-haystack

An integration of Weight and Bias Weave tracer with Haystack

weni-agenteval

A generative AI-powered framework for testing virtual agents.

xai-gpt-shap

A library for explainable AI with SHAP and GPT-based explanations.

ycrash-agent-dev

yCrash Python Agent

yuragi

LLM Confidence Fragility Analyzer — Measure how fragile your AI's confidence really is

yuragi-ai

Reserved name — install the official package: pip install yuragi

zalor-agents

Zalor tracing SDK for AI agents

zipkin-agent

Python Agent for zipkin-agent

zrb-llm-evaluator

Multi-trial experiment runner for zrb chat

vijil-evaluate

Vijil Evaluate review: compare features, pricing, use cases, access model, and alternatives for this Software Testing agent in 2026. Test your agents before you trust your agents

Arize AI

Ship reliable AI agents by monitoring, evaluating, and improving them.

Coval

Automate testing and monitoring for your AI voice and chat agents.

Pezzo

Build, test, and ship AI features faster with Pezzo.

ai-agent-board

Dashboard for managing autonomous AI agent tasks — run with npx agent-board

@asafhm/agentlog

CLI for agentlog — verify and view tamper-evident AI agent run logs

agnostai

Agnost Conversation SDK for TypeScript - Track AI agent conversations with automatic latency measurement

@moda-ai/cli

CLI for Moda - AI agent analytics and observability

trackhound

Sniff out what your AI agent is doing - know exactly what your AI agent did

@andrewhopper/fd

Flight Director — visualize and monitor AI agent state machines

@aws/agent-inspector

A visual inspector for debugging and monitoring AI agents

opik

Opik TypeScript and JavaScript SDK

@flow-a2a/plugin

OpenClaw plugin — Agent communication + LLM cost tracking + telemetry reporting

@elyracode/seo

SEO and LLM optimization for Elyra -- audit, structured data, llms.txt, meta tags

@yfwdecimal/sdk

Agent instrumentation SDK for forensic auditing — auto-signing, hash-chaining, LangChain adapter

@microsoft/applicationinsights-web-basic

Microsoft Application Insights JavaScript SDK - Web Basic

@agentsforge/healthkit

Health monitoring and cluster management for AI agents. Monitor soul drift, manage multi-agent clusters, and ensure agent identity preservation.

@runtime-supervisor/hallucination-eval

Reproducible adversarial eval for AI agents. Deterministic scoring, no model judge. Vibefixing publishes the eval; we don't compete on the leaderboard.

@hackerx333/agenttrace

The open-source circuit breaker and accountability layer for multi-agent AI pipelines. Trace every action. Explain every decision. Short-circuit before the damage propagates.

agent-audit-logger

Lightweight Python middleware that wraps any agent framework and logs every tool call. SOC 2 audit trail for small teams.

agentforge-evidently

Evidently AI metrics + drift hook for AgentForge

langgraph-oep

Capture decision-id-joined evidence from LangGraph agents. Wraps your checkpoint saver to emit OEP tool_permission_packet.v0 records covering the 19 surfaces LangGraph's StateSnapshot does not bind natively.

langgraph-sfm

Causal intent monitoring for LangGraph agents using bundled Structural Final Models.

llmgovernor-helicone

Mirror Helicone traces into LLMGovernor's anomaly engine

llmgovernor-langfuse

Mirror Langfuse traces into LLMGovernor's anomaly engine

llmgovernor-litellm

[DEPRECATED — use llmgovernor[litellm]] LiteLLM callback for LLMGovernor

llmgovernor-otel

OpenTelemetry span exporter for LLMGovernor

uptimes-agents

Uptimes Agents — OpenTelemetry-based observability for AI agents. Three-line init, ships to local uptimes-monitor or direct to api.uptimes.ai.

Phoenix

Open-source tool to trace, evaluate, and fix your AI models.

Vijil

Build, test, and run trusted AI agents that are reliable and safe.

agent-cost-allocator-mcp

Agent Cost Allocator MCP — multi-tenant LLM cost attribution for chargeback billing. Companion to agent-token-budget. Splits one agent's spend across many tenants for invoicing. By MEOK AI Labs.

agent-eval-runner

Run the AI Agent QA Eval Pack against your tool-using LLM agent + grade it with a shareable scorecard badge. Deterministic, OWASP Agentic Top 10 aligned, no LLM-judge.

agentforge-phoenix

Arize Phoenix dashboard hook for AgentForge

agentforge-statsd

StatsD metrics emitter hook for AgentForge

agenticmeter

FinOps for LLM agents: budgets, waste tracking, and caching analysis for OpenAI, Anthropic, and AWS Bedrock

agentrr

Deterministic record-and-replay debugger for AI agent runs

middleware-llmobs

Middleware LLM Observability — OpenInference/Opentelemetry tracing for the Middleware SaaS

AgentOps

Build, debug, and deploy reliable AI agents with full visibility.

@ebb-ai/core

Carbon-aware scheduling for agentic AI workflows — core TypeScript library.

More in Agent frameworks & infrastructure