Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 41 of 57

plott-sdk-langgraph

Plott Analytics SDK for LangGraph - Automatic analytics tracking for LangGraph

polygraphLLM

Hallucination detection package

ppllm

A Python Library to Compute LLM's Perplexity and Surprisal

prefactor-langchain

LangChain integration for Prefactor observability

prism-llm-labs

LLM cost and usage observability SDK

privatetestsetgenerationforLLMeval

A package for generating evaluation set for LLM-based chatbots in a diverse and private manner

probe-llm

Behavioral testing framework for LLMs — test properties, not benchmarks.

proceedgate-crewai

CrewAI integration for ProceedGate cost governance

promotion-harness-agent

Add your description here

promptforge-llmops

A minimalist LLMOps framework for prompt versioning, evaluation and regression testing.

proofagent

pytest for AI agents — test safety, accuracy, tool use, and cost

proofagent-harness

pytest + observability infrastructure for AI agents. Evaluate the agents you build: multi-turn adversarial red teaming and artifact grading with jury based scoring. Observe the agents you use: intelligent risk screening and intent trajectories for coding agents (Claude Code, Cursor). LLM agnostic.

proofagent-sdk

Official Python SDK for ProofAgent™

provedex-langchain

LangChain BaseCallbackHandler that signs every LLM and tool callback via the Provedex sidecar. Covers LangGraph by inheritance.

pruv-crewai

pruv verification layer for CrewAI agents

pruv-langchain

pruv verification layer for LangChain agents

pruv-openai

pruv verification layer for OpenAI Agents SDK

py-llm-shield

Uma biblioteca Python robusta para validar, reparar e extrair dados estruturados de saídas de LLMs usando Pydantic.

pySwarmbytes

UNOFFICIAL Python bindings for Swarmbytes Dashboard API

pydantic-ai-evals

loading...

pydantic-ai-langfuse

Langfuse integration for pydantic-ai

pydantic-ai-mcp-evals

A code-first evaluation framework for testing LLM agents' ability to use MCP tools

pydantic-ai-provenance

Provenance tracking and citation verification for pydantic-ai agents

pydantic-ai-tool-budget

Per-tool budget reminders for Pydantic AI agents

pydantic-ai-visual

pydantic-ai messages visualization

pydantic-llm-tester

A framework for testing LLM performance using pydanticAI

pylibagent

Library for building InfraSonar agents

pyllmtest

The most comprehensive LLM testing framework for Python

pytest-agent-eval

A pytest plugin for LLM evaluation tests with threshold-based pass/fail

pytest-agent-evals

Pytest plugin for evaluating AI Agents

pytest-agent-health

Catch silent agent failures in CI. Behavior lint for LLM agents, powered by agent-failure-debugger.

pytest-agent-observability

pytest plugin that uploads LiveKit-agents eval results to agent-observability

pytest-agentcontract

Deterministic CI tests for LLM agent trajectories — record once, replay offline, assert contracts

pytest-fakellm

Pytest fixtures for the fakellm mock OpenAI/Anthropic server — spin up, reset, and assert with zero boilerplate.

pytest-langchain

Pytest-style test runner for langchain agents

pytest-llm

pytest-llm: A pytest plugin for testing LLM outputs with success rate thresholds.

pytest-llm-assert

Simple LLM-powered assertions for any pytest test

pytest-llm-rubric

A pytest plugin for rubric-based LLM-as-judge testing with auto-discovery and preflight

pytest-llmeval

A pytest plugin to evaluate/benchmark LLM prompts

pytest-llmtest

The pytest for LLMs — fast, Pydantic-based assertions for AI applications

pytest-mockllm

Pytest plugin for mocking OpenAI, Anthropic, Gemini, and LangChain calls

pytest-semantic-llm

Semantic assertions for pytest using LLMs

python-llmexplorer

A Tool that can be used to analyse, monitor prompts usage and LLM ops

qobserva-agent

QObserva agent: decorator-first telemetry for quantum runs (Qiskit, Braket, Cirq, PennyLane, pyQuil, D-Wave).

quark-agent

CLI for Quark-AI — cognitive fingerprinting for AI agents

query-agent-benchmarking

A Python library for benchmarking Weaviate's Query Agent!

quest-llm-judge
r2agent

ClawBench: Can AI Agents Complete Everyday Online Tasks? (alias of claw-bench)

rag-bench

RAG benchmarking tool

rag-bencher

Reproducible RAG baselines and evaluations with LangChain

rag-benchmarking

Framework-agnostic evaluation harness for RAG and agentic AI systems

rag-consistency-scorer

MMD-based semantic consistency scoring for RAG pipeline outputs

rag-contract

Tiny CI tool for RAG retrieval regression testing.

rag-debugger

Real-time debugging SDK for RAG pipelines

rag-doctor

Agentic RAG pipeline failure diagnosis — no database, no LLM API, no cloud

rag-eval

A RAG evaluation framework

rag-eval-kit

Pluggable RAG evaluation framework

rag-eval-pro

Production-grade RAG evaluation library with comprehensive metrics and reporting

rag-eval-tool

A comprehensive evaluation toolkit for RAG and LLMs.

rag-evaluate

RAG Evaluator is a Python library for evaluating Retrieval-Augmented Generation (RAG) systems. It provides various metrics to evaluate the quality of generated text against reference text.

rag-evaluation

A robust Python package for evaluating Retrieval-Augmented Generation (RAG) systems.

rag-evaluator

A library for evaluating Retrieval-Augmented Generation (RAG) systems

rag-exp

A Python package for explaining RAG pipelines.

rag-forge-evaluator

Evaluation engine: RAGAS, DeepEval, LLM-as-Judge, and audit report generation

rag-forge-observability

Observability stack: OpenTelemetry tracing, Langfuse integration, and drift detection

rag-meter

A unified evaluation framework for RAG models.

rag-mongo-logger

Async buffered logger with MongoDB and PostgreSQL support for your RAG applications

rag-oracle

RAG Oracle – failure diagnostics for Retrieval-Augmented Generation systems

rag-pathology

Diagnose RAG pipeline failures by type and location. Four Soils classification. Epistemic mismatch detection.

rag-playbook

Compare RAG patterns with real benchmarks. Run one command, see which pattern wins.

rag-quality-kit

Heuristic quality metrics for RAG retrieval and grounded answers. Python port of @mukundakatta/rag-quality-kit.

rag-sentinel

RAG Evaluation Framework using Ragas metrics and MLflow tracking

rag-tracer

A Python SDK for sending RAG trace node data

raga-llm-eval

Package for LLM Evaluation

raga-llm-hub

Package for LLM Evaluation

raga-testing-platform

Short description or overview of the raga package.

ragaai-catalyst

RAGA AI CATALYST

ragaliq

LLM & RAG evaluation testing framework — hallucination detection, faithfulness metrics, answer relevance scoring, and retrieval pipeline testing with pytest integration

ragang

Retrieval Augmented Generation Assessing tool with No Gold

ragas

Evaluation framework for RAG and LLM applications

ragas-annotator

Experimental Ragas Evaluation UI and Library

ragas-experimental

Experimental extensions for Ragas

ragas-haystack

An integration of Ragas LLM evaluation framework with Haystack

ragas-once

A one-step Ragas cli tool to evaluate RAG apps

ragas-testing

RAG evaluation tool using RAGAS

ragas-wj

ragas for my project

ragbits-evaluate

Evaluation module for Ragbits components

ragcache

3-level LLM cache (exact, semantic, partial) with real-time token cost tracking

ragcheck

Renamed to evalsystem. Please run: pip install evalsystem

ragchecker

RAGChecker: A Fine-grained Framework For Diagnosing Retrieval-Augmented Generation (RAG) systems.

ragcheckerzhongshui

package test

ragcheckerzhongshui0911

package test

ragcheckerzhongshuiv2

package test

ragcheckerzhongshuiv3

package test

ragcraft

Datagen & RagEval for various LLM (Large Language Model) for RAG Apps

ragdebug

Observability SDK for RAG pipelines — auto-trace every stage of your RAG app.

ragdiff

Compare and evaluate RAG systems side-by-side with LLM evaluation. Use as a library or CLI tool.

ragdrift

Silent regression detector for RAG pipelines

ragdrift-py

5-dimensional drift detection for production RAG systems.

rage-toolkit

A framework for retrieval augmented generation evaluation (RAGE)

More in Agent frameworks & infrastructure