Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 32 of 57

agenttrace

A lightweight and hackable tracing/evaluation framework for AI agents and language models

agenttrace-ai

Zero-config local-first visual debugging and auto-evaluation for LLM agents.

agenttrace-core

Core tracing and cost intelligence for AI systems

agenttrace-py

Deterministic execution and replay layer for AI Agents

agenttrace-sdk

AgentTrace SDK - AI Agent Observability & Debugging

agenttracer-ai

Framework-agnostic observability and debugging for AI agents

agenttrail-sdk

AgentTrail Python SDK — observability for agentic AI systems

agentune

Analyze AI agents to understand their performance and get improvement suggestions to make them better

agentune-analyze

Analyze AI agents to understand their performance and get improvement suggestions to make them better

agentune-simulate

Create a customer simulator (twin) based on a set of real conversations.

agentunit

A framework for evaluating, monitoring, and benchmarking multi-agent systems

agentuq

Single-pass runtime reliability instrumentation for LLM agents using token logprobs.

agentux

Portable multi-agent UX stress tester

agentverify

Deterministic testing for AI agents. Assert agent actions, not vibes.

agentvis

Framework-agnostic reasoning trace visualization tool for AI agents.

agentviz

Real-time 3D visualization for multi-agent AI systems

agentwatch-cli

Connect your local Moltbot gateway to AgentWatch cloud

agentwatch-io

Production monitoring SDK for AI agents

agentwatch-monitor

Real-time health and security monitoring for AI coding agents

agentwatch-py

Zero-config OpenAI usage tracking for AgentWatch — prompts never leave your server

agentwatch-sdk

SDK for AgentWatch — observability for multi-agent AI systems

agentwatcher

DevTools overlay for browser-based AI agents — see what your agent is doing in real time.

agentwatchx

Observability SDK for AI Agent Execution — catches hallucinations, silent failures, and missing executions

agentweave-sdk

Observability and mesh layer for multi-agent AI systems — track what your agents decided, why they decided it, and how they're connected.

agentwhisper

A telephone benchmarking system. Measuring information degradation in multi-agent LLM communication chains.

agentwitness

Verifiable evidence for AI-assisted engineering

agentworkbench

Local-first harness for learning, evaluating, comparing, and optimizing repo-native agents.

agentworks

AgentWorks SDK - Instrument and observe multi-agent AI systems

agentx-sdk

The reliability layer for AI agents in production

agentype-cli

Local AI-agent usage analytics and persona archetypes

agentzen

Agent observability, tracing, and evaluation toolkit for agentic AI systems

ai-agent-inspector

Framework-agnostic observability for AI agents

ai-agent-scope

Agent debugging and observability platform - SDK

ai-agent-sentry

Crash reporting for AI agents. Catch failures before your users do.

ai-agent-spend-guardian

Real-time cost monitoring for AI agent deployments across LLM providers

ai-agentree-sdk

Python SDK for AIAgentree — trace, audit, and govern AI agent decisions

ai-agents-metrics

CLI for tracking AI agent task metrics: token cost, retry pressure, and outcome quality.

ai-agenttrace

The accountability layer for AI agents — trace, explain, and circuit-break multi-agent pipelines.

ai-rag-chat-evaluator

Tools for evaluation of RAG Chat Apps using Azure AI Evaluate SDK and OpenAI

aidial-rag-eval

Library for RAG evaluation

aieng-eval-agents

Agentic AI Evaluation reference implementation from Vector Institute AI Engineering

aimon-llamaindex

The AIMon SDK for LlamaIndex related functionalities.

aiqtoolkit-ragaai

Subpackage for RagaAI Catalyst integration in AIQtoolkit

ak-agentbase

Unified AI agent configuration, observability, and model management for multi-agent applications

alibabacloud-paillmtrace20240311

Alibaba Cloud PaiLLMTrace (20240311) SDK Library for Python

aliyun-instrumentation-langchain

Aliyun Langchain Instrumentation

aliyun-instrumentation-llama-index

Aliyun LlamaIndex Instrumentation

aliyun-instrumentation-openai

Aliyun OpanAi Instrumentation

allmetrics

A Unified Python Library for Standardized Metric Evaluation and Robust Data Validation in Machine Learning

alter-ego-llm

Library to run experiments with LLMs

anchor-gpt

Diagnose and find hallucinations in your grounded Large Language Model prompts with Anchor-GPT!

aniket-agentlens-sdk

Python tracing SDK for AgentLens, an open-source observability platform for AI agents.

anosys-claude-code

AnoSys observability hook for Claude Code — automatic audit logging of Claude Code sessions

anosys-logger-4-openai

A helper package, to support implementation to AnoSys platform easly

anosys-logger-4-openai-agents

A helper package for openAi Agents, to support implementation to AnoSys platform easly

anosys-sdk-openai

AnoSys SDK for OpenAI - Automatic instrumentation and logging for OpenAI API calls

anosys-sdk-openai-agents

AnoSys SDK for OpenAI Agents - Automatic instrumentation and logging for OpenAI Agents

anthropic-mock-wrapper

A lightweight wrapper around the Anthropic Python client for testing purposes

anthropic-token-counter

Anthropic Tokens Counter - count tokens for Claude models

anthropic-tokenizer

Fast local Claude tokenizer (Rust + PyO3). Falls back from Anthropic count_tokens API to embedded BPE tokenizer.

any-agent
ao-agent-dev

A development tool for LLM call visualization and debugging

apitracker-agent
apm-agent-utils

This package can help you create Elastic APM instrumentations using regex

argilla-haystack

Argilla-Haystack Integration

argilla-llama-index

Argilla-LlamaIndex Integration

argus-agent-sdk

Python SDK for Argus — AI agent observability and security monitoring

argus-agents

Production readiness platform for AI agent pipelines — detects silent failures, captures full state, enables step-level replay.

argus-llm

Heuristic quality scoring and monitoring for LLM outputs. G-ARVIS scores for Groundedness, Accuracy, Reliability, Variance, Inference Cost, and Safety.

assert-llm-tools

Automated Summary Scoring & Evaluation of Retained Text

assertllm

The pytest for LLMs — fast, Pydantic-based assertions for AI applications

astrago

The Astrago package helps track model training data, allowing you to efficiently collect and manage various metrics and log data generated during the training process.

ateam-llm-tracer

Lightweight tracing SDK for LLM-powered agents with Phoenix integration

ati-integrations-autogen

ATI AutoGen integration: message/tool/turn events -> ATI spans

ati-integrations-autogpt

ATI Integration for AutoGPT

ati-integrations-crewai

ATI CrewAI integration: agent/task lifecycle hooks -> ATI spans

ati-integrations-langchain

ATI LangChain integration: emit agent-aware OpenTelemetry spans via LangChain callbacks

ati-integrations-llamaindex

ATI LlamaIndex integration: retrieval/synthesis callbacks -> ATI spans

atropos-llm

Estimate performance, energy, cost, and break-even tradeoffs for pruning coding LLM deployments.

aumos-agent-eval

Framework for evaluating AI agents across multiple quality dimensions

aumos-agent-observability

OpenTelemetry-native agent tracing, cost attribution, and drift detection

auraone-agent-studio-open

Headless Agent Studio Open protocol, trace-store, sidecar, and export CLI.

auriel-agentlens

An open-source offline AI agent debugging tool: record, replay, analyze failures, and track API costs—free and local.

autogen-primordia

Primordia metering for Microsoft AutoGen

autogenbench

AutoGen Testbed Tools

automlgpt

A Experiment Suite

autonomous-agentpulse

Self-healing telemetry proxy SDK for AI Agents

aws-bedrock-cost-tool

A CLI tool that reports on the cost per model on AWS Bedrock

azure-storage-logging

Logging handlers to send logs to Microsoft Azure Storage

azure-storage-logging-updated

Logging handlers to send logs to Microsoft Azure Storage

azureaicommunity-agent-token-guard

Token usage tracking and quota enforcement middleware for AI agent pipelines

balagan-agent

Chaos engineering framework for AI agents - stress-test agent reliability

ballmatro

A challenging task for LLMs in which they need to create high-scoring Ballatro-like hands

bastion-agent

Error observability for the agent era

bedrock-lens

Real-time token usage and cost monitoring for AWS Bedrock

bedrock-quota

A CLI-based dashboard to see your AWS Bedrock model quotas and usage.

bedrock-tracing

A Python library for tracing Bedrock agent responses with OpenTelemetry.

bedrock-usage-analyzer

Calculate token usage statistics and map quotas for Amazon Bedrock models. Supports both standard AWS regions and AWS GovCloud (US) regions.

bedrockcache

Audit and fix Anthropic prompt caching on AWS Bedrock through any abstraction stack.

bench-my-llm

Dead-simple LLM benchmarking CLI. Measure TTFT, TPS, latency, cost, and quality for any OpenAI-compatible API.

More in Agent frameworks & infrastructure