Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 37 of 57

llm-jury-classifier

Confidence-driven escalation middleware for classifier edge cases

llm-katan

One tiny model, every LLM API. Drop-in test server for OpenAI, Anthropic, Bedrock, and Vertex.

llm-knowledge

Tools to extract, visualize, and quantify latent knowledge in large language models

llm-lab

A minimal, notebook-first framework for running reproducible LLM experiments and comparing multiple models over JSONL/CSV datasets.

llm-leash

Cost ceiling, audit log, and kill switch for LLM agents.

llm-ledger

Production-ready LLM client with audit trail, deterministic caching, and provenance tracking

llm-lens-py

Automatic observability for LLM API calls

llm-locc

Structured output contract testing for LLM systems

llm-logger

Lightweight LLM call logger for OpenAI/Anthropic

llm-logging-debug

Set logging.DEBUG while running LLM

llm-logparser

Privacy-first LLM chat log parser for deterministic local analysis and export

llm-logs-feedback

Feedback on LLM logs (prompts & response)

llm-matched-responses

An llm plugin that provides a matched-responses model for testing - returns configurable responses based on input patterns

llm-message-hash-py

Stable canonical sha256 hash of LLM request/message structures. Recursive key-sorted JSON canonicalization with per-provider presets that drop noise fields. For cache keys and idempotency. Zero runtime deps.

llm-messages-token-helper

A helper library for estimating tokens used by messages.

llm-meter

Accurate LLM usage & cost tracking for Python backends (FastAPI-native)

llm-metrics-lite

A lightweight evaluation framework for large language model outputs

llm-microscope

Official implementation of the LLM-Microscope package functions for the paper

llm-mock

pytest plugin to mock and replay OpenAI and Anthropic API calls — record once, replay forever, no API key needed

llm-mocks

Mock LLMs' responses for unit-tests

llm-model-cost

A package to calculate token costs for LLM models

llm-model-deprecation

Track and check deprecation status of LLM provider models (OpenAI, Anthropic, etc.)

llm-model-diff

Compare LLM model outputs side-by-side with rich diff visualization

llm-monitor

📈 Monitor your LLM integration with Galileo's LLM Monitor!

llm-monitor-sdk

Lightweight SDK for LLM cost & latency monitoring

llm-monitoring-sascha

A small example package

llm-multiple-choice

A Python library for using Large Language Models (LLMs) to fill out multiple-choice questionnaires about chat conversations.

llm-mutation

Mutation testing for LLM prompts. Find the gaps in your eval suite before production does.

llm-obs

Lightweight SDK for LLM inference logging and observability

llm-observatory

Context0 SDK — one import, all calls logged

llm-observe-proxy

OpenAI-compatible LLM proxy with SQLite request capture, observability, and an admin UI.

llm-offset
llm-otel-kit

Drop-in OpenTelemetry GenAI observability for any LLM backend — local or cloud.

llm-path

A tool for tracing LLM requests

llm-perf-tools

Tool for monitoring LLM inference metrics

llm-performance-benchmark

Benchmark the performance (output speed, latency) of OpenAI compatible endpoints

llm-platform-kit

Production-ready LLM platform building blocks — observability, prompts, eval, RAG, hallucination guards

llm-play

a tool that queries LLMs, analyzes responses, and executes experimental pipelines

llm-preflight-check

Local preflight checks for LLM provider keys, endpoints, and model configuration

llm-prettier

A generic JSON prettier and interactive HTML viewer, built especially for parsing embedded JSON from LLM payloads and responses.

llm-price

Pricing + release metadata and cost estimation for LLMs

llm-price-scraper

A simple Python SDK to scrape and retrieve pricing information for Large Language Models (LLMs) from an external webpage, with structured models for easy integration and usage.

llm-pricing

A library for scraping and managing LLM pricing information

llm-probe

Probe LLM models for tool-calling support, think block behavior, and response quality across providers.

llm-profile

LLM benchmarking tools for the LLM CLI

llm-profiler

A brief description of sl-llm

llm-qualitative-sort

LLM-based qualitative sorting using Swiss-system tournament

llm-quick-tools

Prompt evaluation and mock LLM for testing

llm-radar

Real-time observability dashboard for LLM applications. Track prompts, tokens, costs, and latency. One-line integration.

llm-reasoning-tracer

Trace and explain LangChain agent reasoning step-by-step

llm-receipts

LLM Receipts → This package is now `aiir`. Install: pip install aiir

llm-recorder

Record and replay LLM interactions

llm-regression-tester

A flexible library for testing LLM responses against predefined rubrics using OpenAI's API for automated scoring

llm-regtest

Lightweight regression testing for LLM prompts

llm-response-schema-lite-py

Tiny schema validator for structured LLM responses. Python port of @mukundakatta/llm-response-schema-lite.

llm-response-validator

Validate LLM responses against schemas, types, and constraints. Catch bad JSON, missing fields, and hallucinated formats before they crash your app.

llm-rewind

Time-travel debugger for AI agents. Record any production run, replay any failure.

llm-runtime-metrics

Rust-backed performance metrics and request tracing

llm-safecall

Policy checks, schema validation, retries, and logs for Python model calls.

llm-scope-cli

Local debugger for LLM API calls — waterfall breakdown of TTFT, cost, and token speed

llm-scope-observer

Lightweight observability and diagnostics for local and self-hosted LLMs (Ollama, FastAPI, OpenWebUI).I know I messed it up for both of us and I am sorry, she will understand.. you can go ahead and use the library.

llm-sentry

Unified AI reliability platform. One install, 12 diagnostic engines. Continuous monitoring, fault diagnosis, and compliance for LLM pipelines.

llm-sim-eval

Add your description here

llm-similarity-checker

The SimilarityChecker module determines the similarity between two items.

llm-speed

Benchmark any LLM on any hardware. CLI for the llm-speed.com flywheel.

llm-spend-profiler

Track, visualize, and optimize LLM API spending. Two lines of code, zero config.

llm-spend-tracker

Track and limit LLM API spending in real-time. Drop-in middleware for OpenAI, Anthropic, Google, and any OpenAI-compatible API.

llm-stack-watch

A comprehensive LLM tracing and monitoring library

llm-structured-confidence

Extract per-field confidence scores from LLM structured JSON outputs using token-level log-probabilities.

llm-talk

LLM-to-LLM interview framework for evaluating conversational capabilities

llm-telemetry-toolkit

A standardized, file-based telemetry toolkit for LLM Agents.

llm-test-harness

Deterministic testing framework for LLM-powered apps — record/replay cassettes, eval scoring, regression testing

llm-testkit

Professional-grade LLM evaluation framework with ZENO-style HTML reports and enhanced sample analysis

llm-testlab

Comprehensive testing suite for LLM evaluation: hallucination detection, consistency, robustness, safety, and multi-language code generation assessment.

llm-token-guardian

Pre-call cost estimation, tracking, and budgets for OpenAI, Gemini, and Claude

llm-token-tracker

A package to track LLM token usage in context

llm-tokencost

A lightweight Python library for tracking OpenAI and Anthropic SDK costs with budget alerts

llm-tokenmeter

Multi-provider token analysis toolkit for LLMs

llm-tokens-atlas

Open benchmark of LLM tokenization across 5 providers — offline vs empirical calibration deltas.

llm-tokenscope

Profile your LLM payloads. Find the waste. Cut the cost.

llm-toll

Lightweight drop-in Python decorator to track costs, monitor token usage, and enforce budget/rate limits for LLM API calls

llm-tools-trigger-error

Trigger an LLM tools error, useful for testing

llm-trace-sampler-py

Sample LLM traces by risk, errors, latency, and deterministic ids. Python port of @mukundakatta/llm-trace-sampler.

llm-traceguard

End-to-end observability with built-in security guardrails.

llm-tracekit

Meta-package for LLM Tracekit - OpenTelemetry instrumentations for LLM providers.

llm-tracekit-anthropic

OpenTelemetry instrumentation for Anthropic.

llm-tracekit-bedrock

OpenTelemetry instrumentation for AWS Bedrock.

llm-tracekit-core

Core utilities for LLM Tracekit and Coralogix Guardrails SDK.

llm-tracekit-gemini

OpenTelemetry instrumentation for Google Gemini.

llm-tracekit-google-adk

OpenTelemetry instrumentation for Google ADK (Agent Development Kit).

llm-tracekit-langchain

OpenTelemetry instrumentation for LangChain.

llm-tracekit-langgraph

OpenTelemetry instrumentation for LangGraph.

llm-tracekit-litellm

OpenTelemetry instrumentation for LiteLLM.

llm-tracekit-microsoft-foundry

OpenTelemetry instrumentation for Microsoft Foundry (azure-ai-projects).

llm-tracekit-openai

OpenTelemetry instrumentation for OpenAI.

llm-tracekit-openai-agents

OpenTelemetry instrumentation for OpenAI Agents SDK.

llm-tracekit-strands

OpenTelemetry instrumentation for Strands Agents SDK.

llm-usage

LLM usage tracker with cost/latency logging, fallback, and a web dashboard

llm-usage-horizon

Local LLM coding usage collector and Feishu Bitable sync

llm-usage-report

Parse LLM API response logs (Anthropic, OpenAI, Google) and generate token / cost reports. Supports a --alert-at budget alarm that exits non-zero when total cost exceeds a threshold. No framework adoption required.

More in Agent frameworks & infrastructure