Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 33 of 57
BenchGR GPU Benchmark Agent - Test your GPU and submit to the leaderboard
Tool for testing LLMs
A library to benchmark LLMs via their API exposure
Beyond LLM is an toolkit to Build Experiment Evaluate and Observe RAG pipelines
SDK for analyzing and evaluating agent traces stored in BigQuery.
Bisheng Ragas - A RAG evaluation framework compatible with the latest LangChain
Observability logger for Blend360 internal agent workflows.
Trace Haystack LLM and agent calls with Blue Guardrails.
The Forensic Runtime for AI agents - full-spectrum streaming with complete observability
DEPRECATED: LangChain integration is now included in the main braintrust package. Install braintrust instead.
This is a reserved placeholder package to mitigate dependency confusion risks.
This is a reserved placeholder package to mitigate dependency confusion risks.
Python Agent for CA APM
calado LangChain callback handler — captures chain, LLM, tool, and retriever events and ships them to calado for analysis.
Framework for evaluating text and voice AI agents
A light-weight package that allows you to monitor function calls with ease.
The CaptureFlow Tracer is a Python package crafted for in-depth tracing of function calls within Python applications. Its primary function is to capture and relay execution data to the CaptureFlow server-side system for decision making.
centricube ragas modules
SQLite event database for AI agent session tracking and analytics
Quality gates for AI agent swarms: content scoring, hook validation, and agent sync
Validate LLM responses with composable, production-ready checks
A framework for comparing responses from different large language models.
A cost estimation tool for your ChatGPT Plus conversation history.
The most comprehensive LLM testing and evaluation framework for Python.
Pytest for your prompts — test, version and audit LLM outputs in pure Python
Using LLMs for rating items on a scale, comparative judgments, and multiple choice questions.
Find out whether your AI agent's evals are lying to you: stability reports with flip attribution, LLM judge audits, and deterministic checks mined from your KB.
OpenInference DSPy Instrumentation (clank.so fork)
Classifying sentences as AI agent written or not
Compact Claude usage tracking after each Claude task
Track your Claude session + weekly usage. Never waste limits again.
Analyze Claude Code session cache efficiency
Calculates Anthropic Claude cost savings from use of prompt caching
A htop-like TUI for monitoring Claude Code sessions in real-time
Python SDK for Claude Code with Rust core
Claude Code Cache Tail - View Claude Code Cache Hits and Misses
A command-line tool for aggregating and displaying costs associated with Claude API usage
Local web app to explore Claude Code API cost and token usage by day, session, and turn
Terminal dashboard for Claude Code: token usage + agent status with pixel sprites
DEPRECATED: Use llm-interceptor instead. Intercept and analyze LLM traffic from AI coding tools
Drop-in Claude Code Stop hook that ships per-user, per-project usage traces to Langfuse.
Export Claude Code trajectories to Harbor ATIF format
Usage analytics for Claude Code CLI
Real-time token usage visualizer for Claude Code sessions (TUI)
Analytics dashboard for Claude Code sessions
Cross-platform system tray monitor for Claude Code token usage
Comprehensive Streamlit dashboard for Claude Code usage analytics — tokens, costs, tools, sessions, and productivity insights
Rate-limit vitals for Claude Code — burn rate, cache health, and baseline shift detection.
Community Python SDK for the Anthropic Compliance API
Live multi-session context-window meter for Claude Code.
Live terminal dashboard for tracking Claude Code usage and costs
Daily Claude cost analysis with before/after comparison and model health signals
Real-time CLI dashboard for Claude Code tool events
Beautiful HTML visualizations for Claude API Message objects
Display Claude.ai plan usage as ASCII meters in the terminal
Privacy-first Claude Code usage monitor with official limits, state exports, forecasting, and local history
A beautiful, interactive terminal monitor for Claude AI usage — real-time quotas, burn rate, cost analytics, and ML predictions.
claude-monitor SDK — push traces & spans to claude-monitor from Python, wandb-style.
Real-time Claude Pro/Max usage monitor — tray icon, overlay widget, notifications, mini mode, sparklines
Claude Code usage dashboard across multiple devices
Prometheus exporter for Claude API usage and cost metrics
Claude Code efficiency and hygiene toolkit. Surfaces token spend across the 5h / 7d / Sonnet-7d billing windows with per-model / per-agent / per-skill attribution, and audits your effective Claude Code configuration (custom + plugin agents and skills) for overlap and conflicts. Ships an interactive dashboard, a conversational usage-analysis skill, and a structured config-audit skill.
Lightweight status bar monitor for Claude AI token usage
Real-time session tracking and analytics for Claude Code
Trace AI CLI API requests via local reverse and forward proxies. Inspect system prompts, messages, tools, and token usage.
OpenTelemetry instrumentation for Claude agents
Instantly check if your Claude usage is 2x or 1x during Anthropic's March 2026 promotion.
Analyze Claude Code sessions, token usage, tools, and estimated API costs.
CLI to return number of Claude tokens in your text
Real-time token usage monitor for Claude Code
Monitor Claude AI token usage with zero dependencies
Automatic token usage and cost tracking for the Anthropic SDK
Token usage and cost analyzer for Claude Code and OpenClaw session logs
An alternate web-based viewer for logs generated by @mariozechner/claude-trace
Extract and upload Claude Code traces
Transcribe Claude Code JSONL logs to human-readable text
Convert Claude Code JSONL sessions to readable markdown transcripts
macOS menu bar app for Anthropic API usage and cost tracking
Analyze Claude AI usage logs and calculate costs
macOS menu bar app showing Claude API usage limits
Monitor Claude AI token usage with zero dependencies
A real-time terminal monitoring tool for Claude AI token usage
Check your Claude Max/Pro usage limits from the command line
Real-time Claude Code rate limit monitor with system tray app and usage dashboard
Desktop widget and CLI that shows real-time Claude Code usage limits and cost.
YASB widget showing Claude.ai session and weekly usage limits
Claude Code statusline with real subscription usage data
Your history with Claude Code - a Spotify Wrapped style visual terminal experience.
Analytics dashboard for Claude Code power users — mine your conversation history to understand usage patterns, track costs, and optimize your setup.
Claude Usage Monitor TUI - monitor your Claude usage in real-time
Local dashboard for Claude Code token usage, sessions, and cost estimates. Dark UI.
Standardized MLflow tracing setup and CLI for Claude Code, Codex, GitHub Copilot, and Antigravity (agy)
ClawBench: Can AI Agents Complete Everyday Online Tasks? (alias of claw-bench)
Benchmark AI coding agents against each other on the same task
Chutes fast LLM verification.
OpenTelemetry-based observability for Cloudbase Agent Python SDK with OpenInference semantic conventions
CoAgent can help you assess, test, and monitor AI models and agents
Instrumentation package for LangGraph in the Codon ecosystem.