Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 37 of 57
Confidence-driven escalation middleware for classifier edge cases
One tiny model, every LLM API. Drop-in test server for OpenAI, Anthropic, Bedrock, and Vertex.
Tools to extract, visualize, and quantify latent knowledge in large language models
A minimal, notebook-first framework for running reproducible LLM experiments and comparing multiple models over JSONL/CSV datasets.
Cost ceiling, audit log, and kill switch for LLM agents.
Production-ready LLM client with audit trail, deterministic caching, and provenance tracking
Automatic observability for LLM API calls
Structured output contract testing for LLM systems
Lightweight LLM call logger for OpenAI/Anthropic
Set logging.DEBUG while running LLM
Privacy-first LLM chat log parser for deterministic local analysis and export
Feedback on LLM logs (prompts & response)
An llm plugin that provides a matched-responses model for testing - returns configurable responses based on input patterns
Stable canonical sha256 hash of LLM request/message structures. Recursive key-sorted JSON canonicalization with per-provider presets that drop noise fields. For cache keys and idempotency. Zero runtime deps.
A helper library for estimating tokens used by messages.
Accurate LLM usage & cost tracking for Python backends (FastAPI-native)
A lightweight evaluation framework for large language model outputs
Official implementation of the LLM-Microscope package functions for the paper
pytest plugin to mock and replay OpenAI and Anthropic API calls — record once, replay forever, no API key needed
Mock LLMs' responses for unit-tests
A package to calculate token costs for LLM models
Track and check deprecation status of LLM provider models (OpenAI, Anthropic, etc.)
Compare LLM model outputs side-by-side with rich diff visualization
📈 Monitor your LLM integration with Galileo's LLM Monitor!
Lightweight SDK for LLM cost & latency monitoring
A small example package
A Python library for using Large Language Models (LLMs) to fill out multiple-choice questionnaires about chat conversations.
Mutation testing for LLM prompts. Find the gaps in your eval suite before production does.
Lightweight SDK for LLM inference logging and observability
Context0 SDK — one import, all calls logged
OpenAI-compatible LLM proxy with SQLite request capture, observability, and an admin UI.
Drop-in OpenTelemetry GenAI observability for any LLM backend — local or cloud.
A tool for tracing LLM requests
Tool for monitoring LLM inference metrics
Benchmark the performance (output speed, latency) of OpenAI compatible endpoints
Production-ready LLM platform building blocks — observability, prompts, eval, RAG, hallucination guards
a tool that queries LLMs, analyzes responses, and executes experimental pipelines
Local preflight checks for LLM provider keys, endpoints, and model configuration
A generic JSON prettier and interactive HTML viewer, built especially for parsing embedded JSON from LLM payloads and responses.
Pricing + release metadata and cost estimation for LLMs
A simple Python SDK to scrape and retrieve pricing information for Large Language Models (LLMs) from an external webpage, with structured models for easy integration and usage.
A library for scraping and managing LLM pricing information
Probe LLM models for tool-calling support, think block behavior, and response quality across providers.
LLM benchmarking tools for the LLM CLI
A brief description of sl-llm
LLM-based qualitative sorting using Swiss-system tournament
Prompt evaluation and mock LLM for testing
Real-time observability dashboard for LLM applications. Track prompts, tokens, costs, and latency. One-line integration.
Trace and explain LangChain agent reasoning step-by-step
LLM Receipts → This package is now `aiir`. Install: pip install aiir
Record and replay LLM interactions
A flexible library for testing LLM responses against predefined rubrics using OpenAI's API for automated scoring
Lightweight regression testing for LLM prompts
Tiny schema validator for structured LLM responses. Python port of @mukundakatta/llm-response-schema-lite.
Validate LLM responses against schemas, types, and constraints. Catch bad JSON, missing fields, and hallucinated formats before they crash your app.
Time-travel debugger for AI agents. Record any production run, replay any failure.
Rust-backed performance metrics and request tracing
Policy checks, schema validation, retries, and logs for Python model calls.
Local debugger for LLM API calls — waterfall breakdown of TTFT, cost, and token speed
Lightweight observability and diagnostics for local and self-hosted LLMs (Ollama, FastAPI, OpenWebUI).I know I messed it up for both of us and I am sorry, she will understand.. you can go ahead and use the library.
Unified AI reliability platform. One install, 12 diagnostic engines. Continuous monitoring, fault diagnosis, and compliance for LLM pipelines.
Add your description here
The SimilarityChecker module determines the similarity between two items.
Benchmark any LLM on any hardware. CLI for the llm-speed.com flywheel.
Track, visualize, and optimize LLM API spending. Two lines of code, zero config.
Track and limit LLM API spending in real-time. Drop-in middleware for OpenAI, Anthropic, Google, and any OpenAI-compatible API.
A comprehensive LLM tracing and monitoring library
Extract per-field confidence scores from LLM structured JSON outputs using token-level log-probabilities.
LLM-to-LLM interview framework for evaluating conversational capabilities
A standardized, file-based telemetry toolkit for LLM Agents.
Deterministic testing framework for LLM-powered apps — record/replay cassettes, eval scoring, regression testing
Professional-grade LLM evaluation framework with ZENO-style HTML reports and enhanced sample analysis
Comprehensive testing suite for LLM evaluation: hallucination detection, consistency, robustness, safety, and multi-language code generation assessment.
Pre-call cost estimation, tracking, and budgets for OpenAI, Gemini, and Claude
A package to track LLM token usage in context
A lightweight Python library for tracking OpenAI and Anthropic SDK costs with budget alerts
Multi-provider token analysis toolkit for LLMs
Open benchmark of LLM tokenization across 5 providers — offline vs empirical calibration deltas.
Profile your LLM payloads. Find the waste. Cut the cost.
Lightweight drop-in Python decorator to track costs, monitor token usage, and enforce budget/rate limits for LLM API calls
Trigger an LLM tools error, useful for testing
Sample LLM traces by risk, errors, latency, and deterministic ids. Python port of @mukundakatta/llm-trace-sampler.
End-to-end observability with built-in security guardrails.
Meta-package for LLM Tracekit - OpenTelemetry instrumentations for LLM providers.
OpenTelemetry instrumentation for Anthropic.
OpenTelemetry instrumentation for AWS Bedrock.
Core utilities for LLM Tracekit and Coralogix Guardrails SDK.
OpenTelemetry instrumentation for Google Gemini.
OpenTelemetry instrumentation for Google ADK (Agent Development Kit).
OpenTelemetry instrumentation for LangChain.
OpenTelemetry instrumentation for LangGraph.
OpenTelemetry instrumentation for LiteLLM.
OpenTelemetry instrumentation for Microsoft Foundry (azure-ai-projects).
OpenTelemetry instrumentation for OpenAI.
OpenTelemetry instrumentation for OpenAI Agents SDK.
OpenTelemetry instrumentation for Strands Agents SDK.
LLM usage tracker with cost/latency logging, fallback, and a web dashboard
Local LLM coding usage collector and Feishu Bitable sync
Parse LLM API response logs (Anthropic, OpenAI, Google) and generate token / cost reports. Supports a --alert-at budget alarm that exits non-zero when total cost exceeds a threshold. No framework adoption required.