Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 36 of 57
llama-index callbacks langfuse integration
llama-index callbacks literalai integration
llama-index callbacks openinference integration
llama-index instrumentation opentelemetry integration
llama-index callbacks Opik Comet integration
UpTrain Callback for performing evaluations on the LlamaIndex pipeline
llama-index callbacks wandb integration
llama-index evaluation tonic validate integration
Instrumentation and Observability for LlamaIndex
llama-index instrumentation agentops integration
llama-index observability integration with OpenTelemetry
llama-index packs arize_phoenix_query_engine integration
llama-index packs evaluator_benchmarker integration
llama-index packs rag_evaluator integration
llama-index packs trulens_eval_packs integration
Explainable AI with Large Language Models
Llama Stack Evals
Add your description here
Ragas evaluation as an out-of-tree Llama Stack provider
Llama Verifications
A simple token counter for Llama 3.2 models.
Advanced relevance scoring for LLM outputs
Primordia metering for LlamaIndex
Client library for Llama Logs; https://llamalogs.com
LlamaPlastics - A LlamaBench project
opentelemetry instrumentation for openai's completions api
A package for tracking LLM usage and costs
Benchmark autonomous AI agents on task completion, tool use, goal adherence, and safety. Works with any agent — just provide a callable.
Detect silent failures in LLM agents
Deterministic, pytest-style contracts for LLM agent tool calls: order, budgets, forbidden tools, approval gates
Latency and Memory Analysis of Transformer Models for Training and Inference
Runs a throughput benchmark for LLM APIs, measuring generation throughput, prompt throughput, and Time To First Token (TTFT) under various concurrency levels.
Debug your AI programs with ease through a web-based interface. Modify inputs and outputs, and leverage the power of custom Python filters.
LLM Application Performance Monitoring - Real-time monitoring for LLM-powered applications
A/B testing for LLMs. Statistical proof, not vibes.
DEPRECATED: llm-assert has been renamed to callspec. Install with: pip install callspec
A Python package to run LLMAttributor in your computational notebooks.
A placeholder package for llm_autoeval
Behavioral testing for LLM applications. pytest plugin with semantic assertions, multi-turn conversation testing, and drift detection. No LLM judge needed.
Behavioral regression testing tool for LLM model upgrades. Compare model versions and detect behavioral changes.
Evaluate large-language models for undesirable behaviors such as bias.
LLM Benchmarking tool for OLLAMA
A developer-centric CLI tool to systematically evaluate and compare Large Language Models (LLMs)
CLI tool for comparing LLM API pricing, ranked by cost-effectiveness against LMSYS Arena scores
LLM evaluation harness with custom metrics, LLM-as-judge, and regression tracking
LLM Benchmark
LLM Inference Benchmark CLI - measure TTFT, TPS, ITL, E2E latency for any OpenAI-compatible API
Benchmark LLMs with 10 benchmarks & 132K+ questions. 8 providers: OpenAI, Anthropic, Groq, Together, Fireworks, DeepSeek, Ollama, HuggingFace. Unified CLI + Web dashboard.
Benchmark any LLM provider against your actual prompts — latency, cost, quality
Pre-flight LLM cost estimation and budget enforcement
Visualize complex LLM conversation flows in infinite canvases
Calculate CO2 emissions from LLM API calls following Green Software Foundation standards.
A series of coherence and calibration checks for LLMs.
LLM Comparator: An interactive visualization tool for side-by-side LLM evaluation
LLM plugin to compare responses from multiple language models
A Python package for extracting confidence scores from LLM models outputs, particularly using log probabilities
Evaluate LLM consistency under input perturbation using CAT framework metrics
A robust content extractor for LLM outputs with support for JSON, XML, HTML, and code blocks
Define, version, and enforce behavioral contracts on LLM function calls
LLM Output Validation, Linting, and Assertion Layer
Cost tracking for OpenAI, Gemini, and Claude APIs with session management
Real-time cost tracking, budget enforcement, and usage analytics for LLM applications
Estimate LLM request cost and enforce per-request or per-session budgets. Python port of @mukundakatta/llm-cost-guard.
Real-time cost monitoring and budget enforcement for LLM API calls
Python SDK for LLM Cost Monitor - Track, aggregate, and analyze LLM usage costs
A tiny Python library for tracking LLM token usage, latency, and estimated cost - with per-agent-step cost attribution for LangGraph, a local dashboard, and litellm-backed pricing for hundreds of models.
LLM API call token usage-based expense tracker
Track LLM API costs per request. Know where your tokens go.
LLM cost calculator for major providers
A Python package for calculating key metrics to assess LLM performance in various tasks, including extracting structured dataa.
Generate synthetic evaluation datasets from your own documents.
Model-agnostic LLM output degeneration detector — 4-signal composite scoring in a single pass
Transparent, probabilistic classification of text as human-generated or LLM-generated
LLM development debug layer - every API call recorded, nothing lost. Reasoning tokens, prompt diff, session timeline, LangChain/LlamaIndex integration.
A CLI tool for comparing LLM outputs — semantically, visually, and at scale
Extract LLM DNA vectors — low-dimensional representations that capture functional behavior and model evolution.
Production-grade LLM output validation package. Validates LLM outputs using your own LLM when needed — and no LLM when not.
Local-first semantic drift detection for LLM outputs.
Debug plugin for LLM
Carbon-aware scheduling and telemetry for LLM applications
Mathematical health monitor for LLMs — detect degradation and hallucination without NLP
LLM plugin for embedding evaluation with semantic scoring
A comprehensive evaluation toolkit for Large Language Models
Run evals against prompts using LLM
Drop-in observability for LLM applications — quality scoring, hallucination detection, cost tracking, and agent run debugging.
A project for evaluating reasoning capabilities in large language models (LLMs).
placeholder
LLM batch inference latency predictor and profiler CLI tool
LLM Expect is a minimalist, developer-first SDK for testing LLM-powered Python functions using structured JSONL datasets.
Detection and pattern library for LLM agent failures
A Python client for the LLM Feedback API.
Pytest plugin for running non-deterministic LLM tests with automatic retry and beautiful reports
LLM testing framework for validating agent behavior and tool usage
CLI for benchmarking LLM inference servers (vLLM, SGLang, llama.cpp)
A comprehensive toolkit for LLM evaluation, prompt engineering, fine-tuning, and inference optimization
A high-performance instrumentation framework for capturing, streaming, and analyzing internal activations of large language models
A proxy server to intercept and store LLM API calls for fine-tuning dataset collection
Intercept and analyze LLM traffic from AI coding tools
llm's judge each other
A multi-model LLM evaluation framework with consensus-based aggregation for RAG, summarization, and agentic AI workflows