Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 32 of 57
A lightweight and hackable tracing/evaluation framework for AI agents and language models
Zero-config local-first visual debugging and auto-evaluation for LLM agents.
Core tracing and cost intelligence for AI systems
Deterministic execution and replay layer for AI Agents
AgentTrace SDK - AI Agent Observability & Debugging
Framework-agnostic observability and debugging for AI agents
AgentTrail Python SDK — observability for agentic AI systems
Analyze AI agents to understand their performance and get improvement suggestions to make them better
Analyze AI agents to understand their performance and get improvement suggestions to make them better
Create a customer simulator (twin) based on a set of real conversations.
A framework for evaluating, monitoring, and benchmarking multi-agent systems
Single-pass runtime reliability instrumentation for LLM agents using token logprobs.
Portable multi-agent UX stress tester
Deterministic testing for AI agents. Assert agent actions, not vibes.
Framework-agnostic reasoning trace visualization tool for AI agents.
Real-time 3D visualization for multi-agent AI systems
Connect your local Moltbot gateway to AgentWatch cloud
Production monitoring SDK for AI agents
Real-time health and security monitoring for AI coding agents
Zero-config OpenAI usage tracking for AgentWatch — prompts never leave your server
SDK for AgentWatch — observability for multi-agent AI systems
DevTools overlay for browser-based AI agents — see what your agent is doing in real time.
Observability SDK for AI Agent Execution — catches hallucinations, silent failures, and missing executions
Observability and mesh layer for multi-agent AI systems — track what your agents decided, why they decided it, and how they're connected.
A telephone benchmarking system. Measuring information degradation in multi-agent LLM communication chains.
Verifiable evidence for AI-assisted engineering
Local-first harness for learning, evaluating, comparing, and optimizing repo-native agents.
AgentWorks SDK - Instrument and observe multi-agent AI systems
The reliability layer for AI agents in production
Local AI-agent usage analytics and persona archetypes
Agent observability, tracing, and evaluation toolkit for agentic AI systems
Framework-agnostic observability for AI agents
Agent debugging and observability platform - SDK
Crash reporting for AI agents. Catch failures before your users do.
Real-time cost monitoring for AI agent deployments across LLM providers
Python SDK for AIAgentree — trace, audit, and govern AI agent decisions
CLI for tracking AI agent task metrics: token cost, retry pressure, and outcome quality.
The accountability layer for AI agents — trace, explain, and circuit-break multi-agent pipelines.
Tools for evaluation of RAG Chat Apps using Azure AI Evaluate SDK and OpenAI
Library for RAG evaluation
Agentic AI Evaluation reference implementation from Vector Institute AI Engineering
The AIMon SDK for LlamaIndex related functionalities.
Subpackage for RagaAI Catalyst integration in AIQtoolkit
Unified AI agent configuration, observability, and model management for multi-agent applications
Alibaba Cloud PaiLLMTrace (20240311) SDK Library for Python
Aliyun Langchain Instrumentation
Aliyun LlamaIndex Instrumentation
Aliyun OpanAi Instrumentation
A Unified Python Library for Standardized Metric Evaluation and Robust Data Validation in Machine Learning
Library to run experiments with LLMs
Diagnose and find hallucinations in your grounded Large Language Model prompts with Anchor-GPT!
Python tracing SDK for AgentLens, an open-source observability platform for AI agents.
AnoSys observability hook for Claude Code — automatic audit logging of Claude Code sessions
A helper package, to support implementation to AnoSys platform easly
A helper package for openAi Agents, to support implementation to AnoSys platform easly
AnoSys SDK for OpenAI - Automatic instrumentation and logging for OpenAI API calls
AnoSys SDK for OpenAI Agents - Automatic instrumentation and logging for OpenAI Agents
A lightweight wrapper around the Anthropic Python client for testing purposes
Anthropic Tokens Counter - count tokens for Claude models
Fast local Claude tokenizer (Rust + PyO3). Falls back from Anthropic count_tokens API to embedded BPE tokenizer.
A development tool for LLM call visualization and debugging
This package can help you create Elastic APM instrumentations using regex
Argilla-Haystack Integration
Argilla-LlamaIndex Integration
Python SDK for Argus — AI agent observability and security monitoring
Production readiness platform for AI agent pipelines — detects silent failures, captures full state, enables step-level replay.
Heuristic quality scoring and monitoring for LLM outputs. G-ARVIS scores for Groundedness, Accuracy, Reliability, Variance, Inference Cost, and Safety.
Automated Summary Scoring & Evaluation of Retained Text
The pytest for LLMs — fast, Pydantic-based assertions for AI applications
The Astrago package helps track model training data, allowing you to efficiently collect and manage various metrics and log data generated during the training process.
Lightweight tracing SDK for LLM-powered agents with Phoenix integration
ATI AutoGen integration: message/tool/turn events -> ATI spans
ATI Integration for AutoGPT
ATI CrewAI integration: agent/task lifecycle hooks -> ATI spans
ATI LangChain integration: emit agent-aware OpenTelemetry spans via LangChain callbacks
ATI LlamaIndex integration: retrieval/synthesis callbacks -> ATI spans
Estimate performance, energy, cost, and break-even tradeoffs for pruning coding LLM deployments.
Framework for evaluating AI agents across multiple quality dimensions
OpenTelemetry-native agent tracing, cost attribution, and drift detection
Headless Agent Studio Open protocol, trace-store, sidecar, and export CLI.
An open-source offline AI agent debugging tool: record, replay, analyze failures, and track API costs—free and local.
Primordia metering for Microsoft AutoGen
AutoGen Testbed Tools
A Experiment Suite
Self-healing telemetry proxy SDK for AI Agents
A CLI tool that reports on the cost per model on AWS Bedrock
Logging handlers to send logs to Microsoft Azure Storage
Logging handlers to send logs to Microsoft Azure Storage
Token usage tracking and quota enforcement middleware for AI agent pipelines
Chaos engineering framework for AI agents - stress-test agent reliability
A challenging task for LLMs in which they need to create high-scoring Ballatro-like hands
Error observability for the agent era
Real-time token usage and cost monitoring for AWS Bedrock
A CLI-based dashboard to see your AWS Bedrock model quotas and usage.
A Python library for tracing Bedrock agent responses with OpenTelemetry.
Calculate token usage statistics and map quotas for Amazon Bedrock models. Supports both standard AWS regions and AWS GovCloud (US) regions.
Audit and fix Anthropic prompt caching on AWS Bedrock through any abstraction stack.
Dead-simple LLM benchmarking CLI. Measure TTFT, TPS, latency, cost, and quality for any OpenAI-compatible API.