Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 44 of 57
Library to systematically track and evaluate LangGraph based applications.
Library to systematically track and evaluate LLM based applications.
Library to systematically track and evaluate LLM based applications.
Library to systematically track and evaluate LLM based applications.
Library to systematically track and evaluate LLM based applications.
Library to systematically track and evaluate LLM based applications.
Library to systematically track and evaluate LLM based applications.
Library to systematically track and evaluate LLM based applications.
TrustLLM
RAGAS-based evaluation pipeline with trust-specific metrics for TrustRAG
Claim-grounded truth evaluation for LLM outputs: retrieval, per-claim verification, and explicit risk signals
Beautiful terminal UI for monitoring AI agent operations, costs, and attribution
Compare LLM responses side-by-side with local persistence
TuringPulse SDK integration for Anthropic
TuringPulse SDK integration for AWS Bedrock
TuringPulse SDK integration for Cohere
TuringPulse SDK integration for Mistral AI
TuringPulse SDK integration for OpenAI
Safety Agent for evaluating TutorBot AI responses
Panel-based reliability annotation for preference data: measure inter-judge agreement (IJA) with a panel of LLM judges and export TRL-ready DPO data with reliability signals attached.
Umbrage Evals SDK
An integration of UpTrain LLM evaluation framework with Haystack
See where your LLM money goes. Get told exactly how to cut it.
LLM output validation with retry loops and optional Reflexion
Track LLM evaluations to Valohai LLM
Vantage Agent
Record and replay LLM interactions for langchain
Agent observability that traces decisions, not just API calls
VerifAI Python SDK for LLM tracing and evaluation
Tamper-proof audit telemetry for AI agents
LangChain callback handler for Viatoris — emit a signed Viatoris receipt for every LangChain tool call
Visualization tool for various generation tasks on Language Models.
Vital Agent Eval Env
The most comprehensive benchmarking suite for vLLM inference servers
Diagnostic tool for vLLM inference servers
htop-style terminal monitor for vLLM inference servers
LLM-as-a-Judge evaluations for vLLM hosted models
vLLM plugin for interacting with activations during inference
Production-grade vLLM metrics monitoring TUI with persistent storage and Grafana-style visualizations
Comprehensive benchmark suite for semantic router vs direct vLLM evaluation across multiple reasoning datasets
vLLM Semantic Router fleet simulator for capacity planning, SLO validation, and what-if analysis
A monitoring tool for vLLM metrics.
A CLI tool for monitoring and managing VLLM inference servers in Docker containers
Synthetic user research and LLM eval harness — research-grade rigor for developers who can't afford a research team.
Python SDK for VoltAgent - AI Agent Observability and Tracing
Agent Watcher is used to monitor agents running on a VOLTTRON instance
coming soon
coming soon
Read, transform, and write AI agent session logs across formats
Debugger and monitor for AI agents
Unified CLI for the WatchLLM governance ecosystem.
An integration of Weights & Biases Weave tracer with Haystack
An integration of Weight and Bias Weave tracer with Haystack
A generative AI-powered framework for testing virtual agents.
A library for explainable AI with SHAP and GPT-based explanations.
yCrash Python Agent
LLM Confidence Fragility Analyzer — Measure how fragile your AI's confidence really is
Reserved name — install the official package: pip install yuragi
Zalor tracing SDK for AI agents
Python Agent for zipkin-agent
Multi-trial experiment runner for zrb chat
Vijil Evaluate review: compare features, pricing, use cases, access model, and alternatives for this Software Testing agent in 2026. Test your agents before you trust your agents
Ship reliable AI agents by monitoring, evaluating, and improving them.
Automate testing and monitoring for your AI voice and chat agents.
Build, test, and ship AI features faster with Pezzo.
Dashboard for managing autonomous AI agent tasks — run with npx agent-board
CLI for agentlog — verify and view tamper-evident AI agent run logs
Agnost Conversation SDK for TypeScript - Track AI agent conversations with automatic latency measurement
CLI for Moda - AI agent analytics and observability
Sniff out what your AI agent is doing - know exactly what your AI agent did
Flight Director — visualize and monitor AI agent state machines
A visual inspector for debugging and monitoring AI agents
Opik TypeScript and JavaScript SDK
OpenClaw plugin — Agent communication + LLM cost tracking + telemetry reporting
SEO and LLM optimization for Elyra -- audit, structured data, llms.txt, meta tags
Agent instrumentation SDK for forensic auditing — auto-signing, hash-chaining, LangChain adapter
Microsoft Application Insights JavaScript SDK - Web Basic
Health monitoring and cluster management for AI agents. Monitor soul drift, manage multi-agent clusters, and ensure agent identity preservation.
Reproducible adversarial eval for AI agents. Deterministic scoring, no model judge. Vibefixing publishes the eval; we don't compete on the leaderboard.
The open-source circuit breaker and accountability layer for multi-agent AI pipelines. Trace every action. Explain every decision. Short-circuit before the damage propagates.
Lightweight Python middleware that wraps any agent framework and logs every tool call. SOC 2 audit trail for small teams.
Evidently AI metrics + drift hook for AgentForge
Capture decision-id-joined evidence from LangGraph agents. Wraps your checkpoint saver to emit OEP tool_permission_packet.v0 records covering the 19 surfaces LangGraph's StateSnapshot does not bind natively.
Causal intent monitoring for LangGraph agents using bundled Structural Final Models.
Mirror Helicone traces into LLMGovernor's anomaly engine
Mirror Langfuse traces into LLMGovernor's anomaly engine
[DEPRECATED — use llmgovernor[litellm]] LiteLLM callback for LLMGovernor
OpenTelemetry span exporter for LLMGovernor
Uptimes Agents — OpenTelemetry-based observability for AI agents. Three-line init, ships to local uptimes-monitor or direct to api.uptimes.ai.
Open-source tool to trace, evaluate, and fix your AI models.
Build, test, and run trusted AI agents that are reliable and safe.
Agent Cost Allocator MCP — multi-tenant LLM cost attribution for chargeback billing. Companion to agent-token-budget. Splits one agent's spend across many tenants for invoicing. By MEOK AI Labs.
Run the AI Agent QA Eval Pack against your tool-using LLM agent + grade it with a shareable scorecard badge. Deterministic, OWASP Agentic Top 10 aligned, no LLM-judge.
Arize Phoenix dashboard hook for AgentForge
StatsD metrics emitter hook for AgentForge
FinOps for LLM agents: budgets, waste tracking, and caching analysis for OpenAI, Anthropic, and AWS Bedrock
Deterministic record-and-replay debugger for AI agent runs
Middleware LLM Observability — OpenInference/Opentelemetry tracing for the Middleware SaaS
Build, debug, and deploy reliable AI agents with full visibility.
Carbon-aware scheduling for agentic AI workflows — core TypeScript library.