Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 36 of 57

llama-index-callbacks-langfuse

llama-index callbacks langfuse integration

llama-index-callbacks-literalai

llama-index callbacks literalai integration

llama-index-callbacks-openinference

llama-index callbacks openinference integration

llama-index-callbacks-opentelemetry

llama-index instrumentation opentelemetry integration

llama-index-callbacks-opik

llama-index callbacks Opik Comet integration

llama-index-callbacks-uptrain

UpTrain Callback for performing evaluations on the LlamaIndex pipeline

llama-index-callbacks-wandb

llama-index callbacks wandb integration

llama-index-evaluation-tonic-validate

llama-index evaluation tonic validate integration

llama-index-instrumentation

Instrumentation and Observability for LlamaIndex

llama-index-instrumentation-agentops

llama-index instrumentation agentops integration

llama-index-observability-otel

llama-index observability integration with OpenTelemetry

llama-index-packs-arize-phoenix-query-engine

llama-index packs arize_phoenix_query_engine integration

llama-index-packs-evaluator-benchmarker

llama-index packs evaluator_benchmarker integration

llama-index-packs-rag-evaluator

llama-index packs rag_evaluator integration

llama-index-packs-trulens-eval-packs

llama-index packs trulens_eval_packs integration

llama-lime

Explainable AI with Large Language Models

llama-stack-evals

Llama Stack Evals

llama-stack-provider-lmeval

Add your description here

llama-stack-provider-ragas

Ragas evaluation as an out-of-tree Llama Stack provider

llama-verifications

Llama Verifications

llama3.2-token-counter

A simple token counter for Llama 3.2 models.

llamacalc-llamasearch

Advanced relevance scoring for LLM outputs

llamaindex-primordia

Primordia metering for LlamaIndex

llamalogs

Client library for Llama Logs; https://llamalogs.com

llamaplastics-llamasearch

LlamaPlastics - A LlamaBench project

llamatry

opentelemetry instrumentation for openai's completions api

llm-accounting

A package for tracking LLM usage and costs

llm-agent-bench

Benchmark autonomous AI agents on task completion, tool use, goal adherence, and safety. Works with any agent — just provide a callable.

llm-agentguard

Detect silent failures in LLM agents

llm-agentspec

Deterministic, pytest-style contracts for LLM agent tool calls: order, budgets, forbidden tools, approval gates

llm-analysis

Latency and Memory Analysis of Transformer Models for Training and Inference

llm-api-benchmark-mcp-server

Runs a throughput benchmark for LLM APIs, measuring generation throughput, prompt throughput, and Time To First Token (TTFT) under various concurrency levels.

llm-apigateway

Debug your AI programs with ease through a web-based interface. Modify inputs and outputs, and leverage the power of custom Python filters.

llm-apm

LLM Application Performance Monitoring - Real-time monitoring for LLM-powered applications

llm-arbiter

A/B testing for LLMs. Statistical proof, not vibes.

llm-assert

DEPRECATED: llm-assert has been renamed to callspec. Install with: pip install callspec

llm-attributor

A Python package to run LLMAttributor in your computational notebooks.

llm-autoeval

A placeholder package for llm_autoeval

llm-behave

Behavioral testing for LLM applications. pytest plugin with semantic assertions, multi-turn conversation testing, and drift detection. No LLM judge needed.

llm-behavior-diff

Behavioral regression testing tool for LLM model upgrades. Compare model versions and detect behavioral changes.

llm-behavior-eval

Evaluate large-language models for undesirable behaviors such as bias.

llm-bench

LLM Benchmarking tool for OLLAMA

llm-bench-cli

A developer-centric CLI tool to systematically evaluate and compare Large Language Models (LLMs)

llm-bench-cost

CLI tool for comparing LLM API pricing, ranked by cost-effectiveness against LMSYS Arena scores

llm-benchkit

LLM evaluation harness with custom metrics, LLM-as-judge, and regression tracking

llm-benchmark

LLM Benchmark

llm-benchmark-runner

LLM Inference Benchmark CLI - measure TTFT, TPS, ITL, E2E latency for any OpenAI-compatible API

llm-benchmark-toolkit

Benchmark LLMs with 10 benchmarks & 132K+ questions. 8 providers: OpenAI, Anthropic, Groq, Together, Fireworks, DeepSeek, Ollama, HuggingFace. Unified CLI + Web dashboard.

llm-benchmarker

Benchmark any LLM provider against your actual prompts — latency, cost, quality

llm-budget

Pre-flight LLM cost estimation and budget enforcement

llm-canvas

Visualize complex LLM conversation flows in infinite canvases

llm-carbon-calculator

Calculate CO2 emissions from LLM API calls following Green Software Foundation standards.

llm-check

A series of coherence and calibration checks for LLMs.

llm-comparator

LLM Comparator: An interactive visualization tool for side-by-side LLM evaluation

llm-compare

LLM plugin to compare responses from multiple language models

llm-confidence

A Python package for extracting confidence scores from LLM models outputs, particularly using log probabilities

llm-consistency

Evaluate LLM consistency under input perturbation using CAT framework metrics

llm-content-extractor

A robust content extractor for LLM outputs with support for JSON, XML, HTML, and code blocks

llm-contract

Define, version, and enforce behavioral contracts on LLM function calls

llm-contracts

LLM Output Validation, Linting, and Assertion Layer

llm-cost-calculator

Cost tracking for OpenAI, Gemini, and Claude APIs with session management

llm-cost-guard

Real-time cost tracking, budget enforcement, and usage analytics for LLM applications

llm-cost-guard-py

Estimate LLM request cost and enforce per-request or per-session budgets. Python port of @mukundakatta/llm-cost-guard.

llm-cost-guardian

Real-time cost monitoring and budget enforcement for LLM API calls

llm-cost-monitor

Python SDK for LLM Cost Monitor - Track, aggregate, and analyze LLM usage costs

llm-cost-profiler

A tiny Python library for tracking LLM token usage, latency, and estimated cost - with per-agent-step cost attribution for LangGraph, a local dashboard, and litellm-backed pricing for hundreds of models.

llm-cost-tracker

LLM API call token usage-based expense tracker

llm-costlog

Track LLM API costs per request. Know where your tokens go.

llm-costs

LLM cost calculator for major providers

llm-data-lens

A Python package for calculating key metrics to assess LLM performance in various tasks, including extracting structured dataa.

llm-data-simulator

Generate synthetic evaluation datasets from your own documents.

llm-degen-guard

Model-agnostic LLM output degeneration detector — 4-signal composite scoring in a single pass

llm-detector

Transparent, probabilistic classification of text as human-generated or LLM-generated

llm-devproxy

LLM development debug layer - every API call recorded, nothing lost. Reasoning tokens, prompt diff, session timeline, LangChain/LlamaIndex integration.

llm-diff

A CLI tool for comparing LLM outputs — semantically, visually, and at scale

llm-dna

Extract LLM DNA vectors — low-dimensional representations that capture functional behavior and model evolution.

llm-drift-ctl

Production-grade LLM output validation package. Validates LLM outputs using your own LLM when needed — and no LLM when not.

llm-driftscope

Local-first semantic drift detection for LLM outputs.

llm-echo

Debug plugin for LLM

llm-eco-tracker

Carbon-aware scheduling and telemetry for LLM applications

llm-ekg

Mathematical health monitor for LLMs — detect degradation and hallucination without NLP

llm-embedding-eval

LLM plugin for embedding evaluation with semantic scoring

llm-eval-toolkit

A comprehensive evaluation toolkit for Large Language Models

llm-evals-plugin

Run evals against prompts using LLM

llm-evaltrack

Drop-in observability for LLM applications — quality scoring, hallucination detection, cost tracking, and agent run debugging.

llm-evaluation-in-reasoning

A project for evaluating reasoning capabilities in large language models (LLMs).

llm-evaluations

placeholder

llm-execution-time-predictor

LLM batch inference latency predictor and profiler CLI tool

llm-expect

LLM Expect is a minimalist, developer-first SDK for testing LLM-powered Python functions using structured JSONL datasets.

llm-failure-atlas

Detection and pattern library for LLM agent failures

llm-feedback-client

A Python client for the LLM Feedback API.

llm-flaky

Pytest plugin for running non-deterministic LLM tests with automatic retry and beautiful reports

llm-goose

LLM testing framework for validating agent behavior and tool usage

llm-grill

CLI for benchmarking LLM inference servers (vLLM, SGLang, llama.cpp)

llm-insight-forge

A comprehensive toolkit for LLM evaluation, prompt engineering, fine-tuning, and inference optimization

llm-instrumentation

A high-performance instrumentation framework for capturing, streaming, and analyzing internal activations of large language models

llm-intercept

A proxy server to intercept and store LLM API calls for fine-tuning dataset collection

llm-interceptor

Intercept and analyze LLM traffic from AI coding tools

llm-judger

llm's judge each other

llm-jury

A multi-model LLM evaluation framework with consensus-based aggregation for RAG, summarization, and agentic AI workflows

More in Agent frameworks & infrastructure