Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 42 of 57
RAGElo: A Tool for Evaluating Retrieval-Augmented Generation Models
RAG evaluation library — LLM-as-judge for hallucination detection, faithfulness, answer relevancy. Local or self-hosted. Alternative to ragas.
RAG evaluation SDK — RAGの回答品質を3行で計測
A library for evaluating Retrieval-Augmented Generation (RAG) systems
A library for generating dataset and evaluating these datasets on RAG based solutions
Embedding diagnostics toolkit for RAG systems
RAGLens CLI for debugging retrieval behavior in RAG systems
Production-ready RAG pipeline evaluation and observability platform
Monitor your LLM calls. Test your LLM app.
A package for integrating RagMetrics with LLM calls
A modular framework for evaluating and optimizing RAG pipelines.
Local-first evaluation framework for RAG pipelines and AI agents
This library allows for granular testing of llm-applications based on expert input.
Pre-deployment domain difficulty diagnostic for RAG. Know if your benchmark transfers — before you deploy.
Retrieval diagnostics and regression testing for RAG systems.
An evaluation library for RAG models
Lightweight evaluation framework for Retrieval Augmented Generation systems, focused on simplicity and long-term consistency.
This library is to search the best parameters across different steps of the RAG process.
A comprehensive suite for evaluating RAG pipelines.
The Fastest Way to Audit Your RAG - Generate QA datasets & evaluate RAG systems in Colab, Jupyter, or CLI. Privacy-first, any LLM, visual reports.
A tool for evaluating RAG pipelines
A comprehensive Python library for testing and evaluating Retrieval-Augmented Generation (RAG) systems with LLM-generated questions and automated evaluation metrics
Ragtime 🎹 is an LLMOps framework to automatically evaluate Retrieval Augmented Generation (RAG) systems and compare different RAGs / LLMs
Observability and tracing for Retrieval-Augmented Generation pipelines
Lightweight RAG evaluation framework with Korean language support, BGE-M3 embeddings, and HCX-005/Gemini LLMs
A tool for evaluating RAG pipelines
5-dimensional production drift detection for RAG systems.
Decorator-First RAG Visualizer
A Diagnostic View for Explainable and Automated Evaluation of Retrieval-Augmented Generation
OpenTelemetry-native RAG observability SDK with semantic quality scores
A comprehensive benchmarking framework for Retrieval-Augmented Generation (RAG) systems
A package for LLMOps related tasks
Raindrop observability integration for Azure OpenAI
Raindrop observability integration for AWS Bedrock
Raindrop integration for Deep Agents
Raindrop observability integration for DSPy
Raindrop integration for LangChain
Raindrop integration for OpenAI Agents SDK — automatic tracing of agent runs, LLM generations, tool calls, and handoffs.
Raindrop integration for Pydantic AI — automatic observability for Agent run() and run_sync() calls
ReFeR: Improving Evaluation and Reasoning through Hierarchy of Models
REFRACT — Reference-anchored Robust Acid-test for Compressed Transformers. Multi-axis KV-cache fidelity scoring for LLMs across llama.cpp, MLX, vLLM, and SGLang.
Record and replay LLM agent traces for deterministic regression testing — framework-agnostic, pytest-native
SWE-bench for your codebase. Turn merged PRs into reproducible coding-agent benchmarks.
Exporter for Anthropic Agent SDK telemetry to Respan
Respan integration for Haystack pipelines with tracing and logging support
Respan exporter for LiteLLM
Exporter for Respan OpenAI Agents SDK
Respan exporter for Pydantic AI SDK
Respan exporter for Superagent (safety-agent) SDK
Respan instrumentation plugin for AgentSpec
Respan instrumentation plugin for the Anthropic SDK
Respan instrumentation plugin for AutoGen AgentChat
Respan instrumentation plugin for the Claude Agent SDK
Respan instrumentation plugin for CrewAI
Respan instrumentation plugin for DSPy
Respan instrumentation plugin for Haystack
Respan instrumentation plugin for LangChain, LangGraph, and Langflow
Respan instrumentation plugin for LlamaIndex
Respan instrumentation plugin for direct OpenAI SDK usage (native, Traceloop-free)
Respan instrumentation plugin for the OpenAI Agents SDK
Respan instrumentation plugin for PydanticAI
Respan instrumentation plugin for smolagents
Respan instrumentation plugin for Strands Agents
Respan instrumentation plugin for Superagent safety-agent
LLM observability SDK for Revefi - Traceloop-based monitoring with custom ingestor service
A Python library that meters OpenAI usage to Revenium with optional LangChain integration.
Reverie adapter for the OpenAI Agents SDK — captures tracing spans as CognitiveEvents
Chrome DevTools for AI agents — record, inspect, fork, replay, diff.
Lightweight model gateway for capturing LLM call traces during RL agent training
Python package for tracing LLM-based agents
Rollbar server-side agent
Rollbar server-side UDP agent
Add your description here
RukkiePulse agent SDK — auto-instrument Python services and push telemetry to Jaeger
Runtime budget, action, and audit authority for the OpenAI Agents SDK — enforce LLM cost limits, tool call caps, and audit trails before execution.
RunSentry Agent - Monitor and track scheduled job executions
Versatile Python library designed for evaluating the performance of large language models in Natural Language Processing (NLP) tasks. Developed by Sagacify
The Python modules that implements Scalyr Agent 2
Scope agent for Python
('search_llm_actions provides a simple template to collect action trajectories from local (e.g. vllm servers) or remote (e.g. togetherai api) llms by search.',)
Trace storage format based on HDF5 developped by CEA
Splunk SecureApp OpenTelemetry Extension for Python applications
Seer - Multi-Agent System for Evaluating AI Agents
SelfCheckGPT: Assessing text-based responses from LLMs
A Python library for AI governance and experiment tracking that integrates with MLflow
Sentinel — Governable AI Operations Control Plane for LLMOps
Sentinelle monitoring agent for Python apps (Flask, FastAPI, Django)
A texutal interface for monitoring ML experiments in the terminal
Python SDK for Show Holdings AgentOps monitoring
Sidekick Python Debugger
LLMs for unsupervised time series anomaly detection
sik-llm-eval is a simple, yet flexible, framework primarily designed for evaluating Language Model Models (LLMs) on custom use cases.
A module for evaluating language models during training
Tool to monitor openai requests
Thin wrapper around posthog-python's AI integrations. Auto-tags LLM calls with project/service/env for unified cost dashboards.
Anthropic client plugin for space-evals
OpenAI client plugin for space-evals
Distributed LLM evaluation framework for Apache Spark
Speculate is a BDD-style framework for validating LLM behavior, offering rich CLI reporting and support for multiple backends like Ollama, OpenAI, and more.