Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 38 of 57
Drop-in token + cost tracker for OpenAI / LiteLLM / Gemini with caching awareness
Auto-capture OpenAI and Anthropic LLM calls for warehousing and observability
Lightweight LLM observability SDK
LLM Woodpecker - AI Model Debugging Tool with Chat and Raw JSON modes
Evaluation framework for LLM Workers
A real-time visualization and training tool for GPT-2
K9s-inspired observability control tower for LLM & MCP Servers
Multi-threaded explainability for LLMs: words, sentences, documents, images, and tools.
A Python library for estimating the cost of LLM API calls
Identify LLMs by their response fingerprints
A tool designed to evaluate the performance of large language models on mathematical tasks.
A sitreamlined version of "LLM-Math" that judges the math solutions of LLMs.
Package to visualize LLM's Neural Networks activation regions
LLM Judge: simple right/wrong voting across models
LLM as Judge
LLMAuditor is an execution-based evaluation and monitoring framework for LLM, GenAI, and agentic AI applications, providing hallucination detection, cost tracking, and governance auditing.
A framework for load testing llmbench APIs
LLMCache is an open-source caching solution designed to operate seamlessly within your cloud infrastructure.
Calculate LLM token costs from litellm pricing data
Real-time LLM spend enforcement — CLI and tray app
A chess LLM benchmark scored against a local UCI master engine.
CI-native regression testing and migration for LLMs
Research library for black-box experiments on language models.
llmcost
Structured debug snapshots for LLM-assisted debugging
Distributed LLM Evaluation Benchmark Server
Find LLM cost leaks before your bill does. Static analysis for Anthropic and OpenAI client code.
A Flexible Framework for Accelerating LLMs Benchmarking
Reserved name — lleval: LLM eval framework for the FullSense LLM family (on-prem + cloud unified A/B, progressive size matrix, honest disclosure). Reference implementation in development.
A lightweight, cross-platform latency and throughput profiler for LLMs
A metrics and evaluation library for LLMs
Python SDK for the evaluate LLM evaluation framework
LLM evaluation, hallucination detection, AI content detection, compliance, document parsing, governance, security, observability, ground truth testing, conversation evaluation, red team testing, and anomaly detection. 78 metrics across 13 modules. Works with or without API.
Eval
In-process LLM client faker that patches OpenAI, Anthropic, LiteLLM, and LangChain for fast, deterministic testing without network calls
Package to integrate feedback into LLMs
llmfsd: LLM Fake Structured Data, faking Structured Data from any LLM
Semantic caching library for LLM and RAG systems
LLMGovernor — anomaly engine for LLM apps (core SDK with optional LiteLLM bundle)
Open-source LLM evaluation framework — 50+ metrics for RAG, agents, safety, async eval, regression tracking, custom benchmarks, and exportable reports
Release engineering infrastructure for AI behavior - bundle, promote, evaluate, and replay agent behavior artifacts
A package for evaluating language model outputs.
Unified LLM testing lab - test, compare, and interact with multiple AI providers through a single UI
High-performance columnar scan engine for LLM logs stored as JSONL
Automatic cost tracking and observability for LLM applications
Automatic cost tracking and observability for LLM applications
Chaos engineering for LLM apps: mock OpenAI, Anthropic, Gemini and more, break them on demand, and get a verdict on your client's resilience.
Observability and monitoring for AI apps
Reduce LLM costs by 90% - AI recommendations with NO API keys needed!
A CLI tool for LLM performance testing
A package for calculating perplexity using various language models
Fast, offline-first LLM pricing lookup with auto-sync from LiteLLM data. Get token costs, context windows, and capabilities for 2500+ models.
LLM Quality Gate - A provider-agnostic evaluation framework for LLM applications
Intercept OpenAI inputs
Prompt regression testing for RAG pipelines. Like git diff, but for AI.
Deterministic replay debugger for LLM agents
Regression-based LLM benchmark ratings + publication-ready scorecards.
Find the Best Generation Parameters for your LLM & Dataset
Runtime monitor for LLM agent interaction protocols based on session type theory
Comprehensive analysis tool for LLM streaming traffic from PCAP files
Know where your AI money goes. Track LLM API costs per feature, per model, per user.
LLMstudio Tracker is the module of LLMstudio that allows monitoring and logging your LLM calls. By leveraging LLMstudio Tracker, users can gain insights on model performance and streamline development workflows with actionable analytics.
LLM Downstream Task Evaluation Tool.
LLM任务难度评分工具,支持百炼、OpenRouter和Gemini等大模型
llmtest package
Anthropic provider for llmtest
CLI for llmtest
Core library for llmtest — Pydantic models, assertions, runners
pytest for LLM apps - test for grounding failures, prompt injection, safety, and regressions
Ollama provider for llmtest
OpenAI provider for llmtest
Production-quality performance validation and regression testing for LLM inference systems
pytest plugin for llmtest
A new package that helps developers integration-test AI and LLM applications by validating structured outputs. It takes a user's test scenario or prompt as input, sends it to an LLM, and uses pattern
LLM Tests
A framework for evaluating overthinking and basic reasoning capabilities of Large Language Models
Count tokens and explore chat templates using HuggingFace tokenizers or OpenAI tiktoken
LLM-powered system monitoring with real-time performance insights
A lightweight LLM observability and evaluation framework.
Zero-config decorator for tracing LLM calls
Python bindings for LLMTrace — security-aware LLM observability
Lightweight structured tracing for LLM applications
LLMTracer: A tool to trace the execution of a program that uses LLMs and visualize the results.
Auto-track LLM cost, latency, and usage. Two lines of code, every provider.
End-to-end observability with built-in security guardrails.
A tool to view LLM call traces from JSONL log files.
Lightweight LLM observability. SQLite-based tracing with zero infrastructure.
Python bindings for LLMTrace — security-aware LLM observability
A concise description of your package.
Eval-Scope: Lightweight LLMs Evaluation Framework
Oncoshot LLM validation framework
Record and replay LLM API responses for fast, free, deterministic tests
LLM Hallucination & Drift Detection — Verify LLM outputs for accuracy, consistency, and reliability