Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 41 of 57
Plott Analytics SDK for LangGraph - Automatic analytics tracking for LangGraph
Hallucination detection package
A Python Library to Compute LLM's Perplexity and Surprisal
LangChain integration for Prefactor observability
LLM cost and usage observability SDK
A package for generating evaluation set for LLM-based chatbots in a diverse and private manner
Behavioral testing framework for LLMs — test properties, not benchmarks.
CrewAI integration for ProceedGate cost governance
Add your description here
A minimalist LLMOps framework for prompt versioning, evaluation and regression testing.
pytest for AI agents — test safety, accuracy, tool use, and cost
pytest + observability infrastructure for AI agents. Evaluate the agents you build: multi-turn adversarial red teaming and artifact grading with jury based scoring. Observe the agents you use: intelligent risk screening and intent trajectories for coding agents (Claude Code, Cursor). LLM agnostic.
Official Python SDK for ProofAgent™
LangChain BaseCallbackHandler that signs every LLM and tool callback via the Provedex sidecar. Covers LangGraph by inheritance.
pruv verification layer for CrewAI agents
pruv verification layer for LangChain agents
pruv verification layer for OpenAI Agents SDK
Uma biblioteca Python robusta para validar, reparar e extrair dados estruturados de saídas de LLMs usando Pydantic.
UNOFFICIAL Python bindings for Swarmbytes Dashboard API
loading...
Langfuse integration for pydantic-ai
A code-first evaluation framework for testing LLM agents' ability to use MCP tools
Provenance tracking and citation verification for pydantic-ai agents
Per-tool budget reminders for Pydantic AI agents
pydantic-ai messages visualization
A framework for testing LLM performance using pydanticAI
Library for building InfraSonar agents
The most comprehensive LLM testing framework for Python
A pytest plugin for LLM evaluation tests with threshold-based pass/fail
Pytest plugin for evaluating AI Agents
Catch silent agent failures in CI. Behavior lint for LLM agents, powered by agent-failure-debugger.
pytest plugin that uploads LiveKit-agents eval results to agent-observability
Deterministic CI tests for LLM agent trajectories — record once, replay offline, assert contracts
Pytest fixtures for the fakellm mock OpenAI/Anthropic server — spin up, reset, and assert with zero boilerplate.
Pytest-style test runner for langchain agents
pytest-llm: A pytest plugin for testing LLM outputs with success rate thresholds.
Simple LLM-powered assertions for any pytest test
A pytest plugin for rubric-based LLM-as-judge testing with auto-discovery and preflight
A pytest plugin to evaluate/benchmark LLM prompts
The pytest for LLMs — fast, Pydantic-based assertions for AI applications
Pytest plugin for mocking OpenAI, Anthropic, Gemini, and LangChain calls
Semantic assertions for pytest using LLMs
A Tool that can be used to analyse, monitor prompts usage and LLM ops
QObserva agent: decorator-first telemetry for quantum runs (Qiskit, Braket, Cirq, PennyLane, pyQuil, D-Wave).
CLI for Quark-AI — cognitive fingerprinting for AI agents
A Python library for benchmarking Weaviate's Query Agent!
ClawBench: Can AI Agents Complete Everyday Online Tasks? (alias of claw-bench)
RAG benchmarking tool
Reproducible RAG baselines and evaluations with LangChain
Framework-agnostic evaluation harness for RAG and agentic AI systems
MMD-based semantic consistency scoring for RAG pipeline outputs
Tiny CI tool for RAG retrieval regression testing.
Real-time debugging SDK for RAG pipelines
Agentic RAG pipeline failure diagnosis — no database, no LLM API, no cloud
A RAG evaluation framework
Pluggable RAG evaluation framework
Production-grade RAG evaluation library with comprehensive metrics and reporting
A comprehensive evaluation toolkit for RAG and LLMs.
RAG Evaluator is a Python library for evaluating Retrieval-Augmented Generation (RAG) systems. It provides various metrics to evaluate the quality of generated text against reference text.
A robust Python package for evaluating Retrieval-Augmented Generation (RAG) systems.
A library for evaluating Retrieval-Augmented Generation (RAG) systems
A Python package for explaining RAG pipelines.
Evaluation engine: RAGAS, DeepEval, LLM-as-Judge, and audit report generation
Observability stack: OpenTelemetry tracing, Langfuse integration, and drift detection
A unified evaluation framework for RAG models.
Async buffered logger with MongoDB and PostgreSQL support for your RAG applications
RAG Oracle – failure diagnostics for Retrieval-Augmented Generation systems
Diagnose RAG pipeline failures by type and location. Four Soils classification. Epistemic mismatch detection.
Compare RAG patterns with real benchmarks. Run one command, see which pattern wins.
Heuristic quality metrics for RAG retrieval and grounded answers. Python port of @mukundakatta/rag-quality-kit.
RAG Evaluation Framework using Ragas metrics and MLflow tracking
A Python SDK for sending RAG trace node data
Package for LLM Evaluation
Package for LLM Evaluation
Short description or overview of the raga package.
RAGA AI CATALYST
LLM & RAG evaluation testing framework — hallucination detection, faithfulness metrics, answer relevance scoring, and retrieval pipeline testing with pytest integration
Retrieval Augmented Generation Assessing tool with No Gold
Evaluation framework for RAG and LLM applications
Experimental Ragas Evaluation UI and Library
Experimental extensions for Ragas
An integration of Ragas LLM evaluation framework with Haystack
A one-step Ragas cli tool to evaluate RAG apps
RAG evaluation tool using RAGAS
ragas for my project
Evaluation module for Ragbits components
3-level LLM cache (exact, semantic, partial) with real-time token cost tracking
Renamed to evalsystem. Please run: pip install evalsystem
RAGChecker: A Fine-grained Framework For Diagnosing Retrieval-Augmented Generation (RAG) systems.
package test
package test
package test
package test
Datagen & RagEval for various LLM (Large Language Model) for RAG Apps
Observability SDK for RAG pipelines — auto-trace every stage of your RAG app.
Compare and evaluate RAG systems side-by-side with LLM evaluation. Use as a library or CLI tool.
Silent regression detector for RAG pipelines
5-dimensional drift detection for production RAG systems.
A framework for retrieval augmented generation evaluation (RAGE)