Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 42 of 57

ragelo

RAGElo: A Tool for Evaluating Retrieval-Augmented Generation Models

rageval-ai

RAG evaluation library — LLM-as-judge for hallucination detection, faithfulness, answer relevancy. Local or self-hosted. Alternative to ragas.

rageval-sdk

RAG evaluation SDK — RAGの回答品質を3行で計測

ragevals

A library for evaluating Retrieval-Augmented Generation (RAG) systems

ragformance

A library for generating dataset and evaluating these datasets on RAG based solutions

raglens

Embedding diagnostics toolkit for RAG systems

raglens-cli

RAGLens CLI for debugging retrieval behavior in RAG systems

raglint

Production-ready RAG pipeline evaluation and observability platform

ragmetrics-client

Monitor your LLM calls. Test your LLM app.

ragmetrics-pkg

A package for integrating RagMetrics with LLM calls

ragmint

A modular framework for evaluating and optimizing RAG pipelines.

ragnarok-ai

Local-first evaluation framework for RAG pipelines and AI agents

ragpill

This library allows for granular testing of llm-applications based on expert input.

ragprobe

Pre-deployment domain difficulty diagnostic for RAG. Know if your benchmark transfers — before you deploy.

ragprobe-diagnostics

Retrieval diagnostics and regression testing for RAG systems.

ragrank

An evaluation library for RAG models

ragret

Lightweight evaluation framework for Retrieval Augmented Generation systems, focused on simplicity and long-term consistency.

ragrid

This library is to search the best parameters across different steps of the RAG process.

ragscope

A comprehensive suite for evaluating RAG pipelines.

ragscore

The Fastest Way to Audit Your RAG - Generate QA datasets & evaluate RAG systems in Colab, Jupyter, or CLI. Privacy-first, any LLM, visual reports.

ragstack-ai-ragulate

A tool for evaluating RAG pipelines

ragtester

A comprehensive Python library for testing and evaluating Retrieval-Augmented Generation (RAG) systems with LLM-generated questions and automated evaluation metrics

ragtime

Ragtime 🎹 is an LLMOps framework to automatically evaluate Retrieval Augmented Generation (RAG) systems and compare different RAGs / LLMs

ragtrace

Observability and tracing for Retrieval-Augmented Generation pipelines

ragtrace-lite

Lightweight RAG evaluation framework with Korean language support, BGE-M3 embeddings, and HCX-005/Gemini LLMs

ragulate

A tool for evaluating RAG pipelines

ragvitals

5-dimensional production drift detection for RAG systems.

ragviz

Decorator-First RAG Visualizer

ragvue

A Diagnostic View for Explainable and Automated Evaluation of Retrieval-Augmented Generation

ragwatch

OpenTelemetry-native RAG observability SDK with semantic quality scores

ragworkbench

A comprehensive benchmarking framework for Retrieval-Augmented Generation (RAG) systems

ragwrapper

A package for LLMOps related tasks

raindrop-azure-openai

Raindrop observability integration for Azure OpenAI

raindrop-bedrock

Raindrop observability integration for AWS Bedrock

raindrop-deep-agents

Raindrop integration for Deep Agents

raindrop-dspy

Raindrop observability integration for DSPy

raindrop-langchain

Raindrop integration for LangChain

raindrop-openai-agents

Raindrop integration for OpenAI Agents SDK — automatic tracing of agent runs, LLM generations, tool calls, and handoffs.

raindrop-pydantic-ai

Raindrop integration for Pydantic AI — automatic observability for Agent run() and run_sync() calls

realtimex-any-agent
refer-agents

ReFeR: Improving Evaluation and Reasoning through Hierarchy of Models

refract-llm

REFRACT — Reference-anchored Robust Acid-test for Compressed Transformers. Multi-axis KV-cache fidelity scoring for LLMs across llama.cpp, MLX, vLLM, and SGLang.

replay-agent

Record and replay LLM agent traces for deterministic regression testing — framework-agnostic, pytest-native

repoagentbench

SWE-bench for your codebase. Turn merged PRs into reproducible coding-agent benchmarks.

respan-exporter-anthropic-agents

Exporter for Anthropic Agent SDK telemetry to Respan

respan-exporter-haystack

Respan integration for Haystack pipelines with tracing and logging support

respan-exporter-litellm

Respan exporter for LiteLLM

respan-exporter-openai-agents

Exporter for Respan OpenAI Agents SDK

respan-exporter-pydantic-ai

Respan exporter for Pydantic AI SDK

respan-exporter-superagent

Respan exporter for Superagent (safety-agent) SDK

respan-instrumentation-agentspec

Respan instrumentation plugin for AgentSpec

respan-instrumentation-anthropic

Respan instrumentation plugin for the Anthropic SDK

respan-instrumentation-autogen

Respan instrumentation plugin for AutoGen AgentChat

respan-instrumentation-claude-agent-sdk

Respan instrumentation plugin for the Claude Agent SDK

respan-instrumentation-crewai

Respan instrumentation plugin for CrewAI

respan-instrumentation-dspy

Respan instrumentation plugin for DSPy

respan-instrumentation-haystack

Respan instrumentation plugin for Haystack

respan-instrumentation-langchain

Respan instrumentation plugin for LangChain, LangGraph, and Langflow

respan-instrumentation-llama-index

Respan instrumentation plugin for LlamaIndex

respan-instrumentation-openai

Respan instrumentation plugin for direct OpenAI SDK usage (native, Traceloop-free)

respan-instrumentation-openai-agents

Respan instrumentation plugin for the OpenAI Agents SDK

respan-instrumentation-pydantic-ai

Respan instrumentation plugin for PydanticAI

respan-instrumentation-smolagents

Respan instrumentation plugin for smolagents

respan-instrumentation-strands-agents

Respan instrumentation plugin for Strands Agents

respan-instrumentation-superagent

Respan instrumentation plugin for Superagent safety-agent

revefi-llm-sdk

LLM observability SDK for Revefi - Traceloop-based monitoring with custom ingestor service

revenium-middleware-openai

A Python library that meters OpenAI usage to Revenium with optional LangChain integration.

reverie-adapter-openai

Reverie adapter for the OpenAI Agents SDK — captures tracing spans as CognitiveEvents

rewind-agent

Chrome DevTools for AI agents — record, inspect, fork, replay, diff.

rllm-model-gateway

Lightweight model gateway for capturing LLM call traces during RL agent training

robonito-agent-tracing-sdk

Python package for tracing LLM-based agents

rollbar-agent

Rollbar server-side agent

rollbar-udp-agent

Rollbar server-side UDP agent

rouge4llm

Add your description here

rukkie-agent

RukkiePulse agent SDK — auto-instrument Python services and push telemetry to Jaeger

runcycles-openai-agents

Runtime budget, action, and audit authority for the OpenAI Agents SDK — enforce LLM cost limits, tool call caps, and audit trails before execution.

runsentry-agent

RunSentry Agent - Monitor and track scheduled job executions

saga-llm-evaluation

Versatile Python library designed for evaluating the performance of large language models in Natural Language Processing (NLP) tasks. Developed by Sagacify

scalyr-agent-2

The Python modules that implements Scalyr Agent 2

scopeagent

Scope agent for Python

search-llm-actions

('search_llm_actions provides a simple template to collect action trajectories from local (e.g. vllm servers) or remote (e.g. togetherai api) llms by search.',)

secbench-storage

Trace storage format based on HDF5 developped by CEA

secureapp-python-agent

Splunk SecureApp OpenTelemetry Extension for Python applications

seeragents

Seer - Multi-Agent System for Evaluating AI Agents

selfcheckgpt

SelfCheckGPT: Assessing text-based responses from LLMs

semantix-agentix

A Python library for AI governance and experiment tracking that integrates with MLflow

sentinel-llmops

Sentinel — Governable AI Operations Control Plane for LLMOps

sentinelle-agent

Sentinelle monitoring agent for Python apps (Flask, FastAPI, Django)

shellml

A texutal interface for monitoring ML experiments in the terminal

showholdings-agentmon

Python SDK for Show Holdings AgentOps monitoring

sidekick-agent-python

Sidekick Python Debugger

sigllm

LLMs for unsupervised time series anomaly detection

sik-llm-eval

sik-llm-eval is a simple, yet flexible, framework primarily designed for evaluating Language Model Models (LLMs) on custom use cases.

sinhala-llm-evaluator

A module for evaluating language models during training

site24x7-openai-observability

Tool to monitor openai requests

sotalab-llm-metrics

Thin wrapper around posthog-python's AI integrations. Auto-tags LLM calls with project/service/env for unified cost dashboards.

space-evals-client-anthropic

Anthropic client plugin for space-evals

space-evals-client-openai

OpenAI client plugin for space-evals

spark-llm-eval

Distributed LLM evaluation framework for Apache Spark

speculate-llm

Speculate is a BDD-style framework for validating LLM behavior, offering rich CLI reporting and support for multiple backends like Ollama, OpenAI, and more.

More in Agent frameworks & infrastructure