Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 33 of 57

benchgr-agent

BenchGR GPU Benchmark Agent - Test your GPU and submit to the leaderboard

benchllm

Tool for testing LLMs

benchmark-llm-serving

A library to benchmark LLMs via their API exposure

beyondllm

Beyond LLM is an toolkit to Build Experiment Evaluate and Observe RAG pipelines

bigquery-agent-analytics

SDK for analyzing and evaluating agent traces stored in BigQuery.

bisheng-ragas
bisheng-ragas-lc1

Bisheng Ragas - A RAG evaluation framework compatible with the latest LangChain

blend-agents-observability

Observability logger for Blend360 internal agent workflows.

blueguardrails-haystack

Trace Haystack LLM and agent calls with Blue Guardrails.

bond-agent

The Forensic Runtime for AI agents - full-spectrum streaming with complete observability

botrun-llm-ranking
braintrust-langchain

DEPRECATED: LangChain integration is now included in the main braintrust package. Install braintrust instead.

bytedance-xgpt-evals

This is a reserved placeholder package to mitigate dependency confusion risks.

bytedweave-trace-analyze-agent

This is a reserved placeholder package to mitigate dependency confusion risks.

ca-apm-agent

Python Agent for CA APM

calado-langchain

calado LangChain callback handler — captures chain, LLM, tool, and retriever events and ships them to calado for analysis.

calibrate-agent

Framework for evaluating text and voice AI agents

callmonitor

A light-weight package that allows you to monitor function calls with ease.

captureflow-agent

The CaptureFlow Tracer is a Python package crafted for in-depth tracing of function calls within Python applications. Its primary function is to capture and relay execution data to the CaptureFlow server-side system for decision making.

centricube-ragas

centricube ragas modules

cervellaswarm-event-store

SQLite event database for AI agent session tracking and analytics

cervellaswarm-quality-gates

Quality gates for AI agent swarms: content scoring, hook validation, and agent sync

chaincheck-llm

Validate LLM responses with composable, production-ready checks

chat2llms

A framework for comparing responses from different large language models.

chatgpt-cost-estimator

A cost estimation tool for your ChatGPT Plus conversation history.

checkllm

The most comprehensive LLM testing and evaluation framework for Python.

chitragupta

Pytest for your prompts — test, version and audit LLM outputs in pure Python

choicellm

Using LLMs for rating items on a scale, comparative judgments, and multiple choice questions.

ciagent

Find out whether your AI agent's evals are lying to you: stability reports with flip attribution, LLM judge audits, and deterministic checks mined from your KB.

clank-so-openinference-instrumentation-dspy

OpenInference DSPy Instrumentation (clank.so fork)

classifygpt

Classifying sentences as AI agent written or not

claude-burn

Compact Claude usage tracking after each Claude task

claude-burnrate

Track your Claude session + weekly usage. Never waste limits again.

claude-cache-analyzer

Analyze Claude Code session cache efficiency

claude-cache-savings

Calculates Anthropic Claude cost savings from use of prompt caching

claude-cctop

A htop-like TUI for monitoring Claude Code sessions in real-time

claude-code-analytics

Python SDK for Claude Code with Rust core

claude-code-cache-tail

Claude Code Cache Tail - View Claude Code Cache Hits and Misses

claude-code-cost-collector

A command-line tool for aggregating and displaying costs associated with Claude API usage

claude-code-cost-explorer

Local web app to explore Claude Code API cost and token usage by day, session, and turn

claude-code-dashboard

Terminal dashboard for Claude Code: token usage + agent status with pixel sprites

claude-code-inspector

DEPRECATED: Use llm-interceptor instead. Intercept and analyze LLM traffic from AI coding tools

claude-code-langfuse-hook

Drop-in Claude Code Stop hook that ships per-user, per-project usage traces to Langfuse.

claude-code-logger

Export Claude Code trajectories to Harbor ATIF format

claude-code-stats

Usage analytics for Claude Code CLI

claude-code-token-visualizer

Real-time token usage visualizer for Claude Code sessions (TUI)

claude-code-tracer

Analytics dashboard for Claude Code sessions

claude-code-usage

Cross-platform system tray monitor for Claude Code token usage

claude-code-visibility

Comprehensive Streamlit dashboard for Claude Code usage analytics — tokens, costs, tools, sessions, and productivity insights

claude-code-vitals

Rate-limit vitals for Claude Code — burn rate, cache health, and baseline shift detection.

claude-compliance-sdk

Community Python SDK for the Anthropic Compliance API

claude-context-bar

Live multi-session context-window meter for Claude Code.

claude-cost

Live terminal dashboard for tracking Claude Code usage and costs

claude-cost-compare

Daily Claude cost analysis with before/after comparison and model health signals

claude-dashboard

Real-time CLI dashboard for Claude Code tool events

claude-message-visualizer

Beautiful HTML visualizations for Claude API Message objects

claude-meter

Display Claude.ai plan usage as ASCII meters in the terminal

claude-monitor

Privacy-first Claude Code usage monitor with official limits, state exports, forecasting, and local history

claude-monitor-cli

A beautiful, interactive terminal monitor for Claude AI usage — real-time quotas, burn rate, cost analytics, and ML predictions.

claude-monitor-sdk

claude-monitor SDK — push traces & spans to claude-monitor from Python, wandb-style.

claude-monitor-usage

Real-time Claude Pro/Max usage monitor — tray icon, overlay widget, notifications, mini mode, sparklines

claude-multi-usage

Claude Code usage dashboard across multiple devices

claude-prometheus-exporter

Prometheus exporter for Claude API usage and cost metrics

claude-prospector

Claude Code efficiency and hygiene toolkit. Surfaces token spend across the 5h / 7d / Sonnet-7d billing windows with per-model / per-agent / per-skill attribution, and audits your effective Claude Code configuration (custom + plugin agents and skills) for overlap and conflicts. Ships an interactive dashboard, a conversational usage-analysis skill, and a structured config-audit skill.

claude-statusbar

Lightweight status bar monitor for Claude AI token usage

claude-statusline

Real-time session tracking and analytics for Claude Code

claude-tap

Trace AI CLI API requests via local reverse and forward proxies. Inspect system prompts, messages, tools, and token usage.

claude-telemetry

OpenTelemetry instrumentation for Claude agents

claude-throttle

Instantly check if your Claude usage is 2x or 1x during Anthropic's March 2026 promotion.

claude-times

Analyze Claude Code sessions, token usage, tools, and estimated API costs.

claude-token-count

CLI to return number of Claude tokens in your text

claude-token-dashboard

Real-time token usage monitor for Claude Code

claude-token-monitor

Monitor Claude AI token usage with zero dependencies

claude-token-tracker

Automatic token usage and cost tracking for the Anthropic SDK

claude-tokens

Token usage and cost analyzer for Claude Code and OpenClaw session logs

claude-trace-viewer

An alternate web-based viewer for logs generated by @mariozechner/claude-trace

claude-traces-client

Extract and upload Claude Code traces

claude-transcriber

Transcribe Claude Code JSONL logs to human-readable text

claude-transcript

Convert Claude Code JSONL sessions to readable markdown transcripts

claude-usage

macOS menu bar app for Anthropic API usage and cost tracking

claude-usage-analyzer

Analyze Claude AI usage logs and calculate costs

claude-usage-bar

macOS menu bar app showing Claude API usage limits

claude-usage-cli

Monitor Claude AI token usage with zero dependencies

claude-usage-monitor

A real-time terminal monitoring tool for Claude AI token usage

claude-usage-plz

Check your Claude Max/Pro usage limits from the command line

claude-usage-tray

Real-time Claude Code rate limit monitor with system tray app and usage dashboard

claude-usage-widget

Desktop widget and CLI that shows real-time Claude Code usage limits and cost.

claude-usage-yasb

YASB widget showing Claude.ai session and weekly usage limits

claude-vibeline

Claude Code statusline with real subscription usage data

claude-wrapped

Your history with Claude Code - a Spotify Wrapped style visual terminal experience.

claudealytics

Analytics dashboard for Claude Code power users — mine your conversation history to understand usage patterns, track costs, and optimize your setup.

claudemon

Claude Usage Monitor TUI - monitor your Claude usage in real-time

claudepulse

Local dashboard for Claude Code token usage, sessions, and cost estimates. Dark UI.

claudetracing

Standardized MLflow tracing setup and CLI for Claude Code, Codex, GitHub Copilot, and Antigravity (agy)

claw-agent

ClawBench: Can AI Agents Complete Everyday Online Tasks? (alias of claw-bench)

cli-agent-bench

Benchmark AI coding agents against each other on the same task

cllmv

Chutes fast LLM verification.

cloudbase-agent-observability

OpenTelemetry-based observability for Cloudbase Agent Python SDK with OpenInference semantic conventions

coa-dev-coagent

CoAgent can help you assess, test, and monitor AI models and agents

codon-instrumentation-langgraph

Instrumentation package for LangGraph in the Codon ecosystem.

More in Agent frameworks & infrastructure