Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 34 of 57
Pluggable agent backends for RAGAS dataset generation (default Llama Stack Responses API or LangGraph)
AutoGen 0.4+ integration for the Colber platform — per-tool observability, AutoGen Memory adapter, and a 5-service toolkit of typed BaseTool[Args, Result] subclasses.
CrewAI integration for the Colber platform — step/task callbacks, long-term memory, and a 5-service toolkit.
LangChain integration for the Colber platform — callbacks, memory, and a 6-service toolkit.
Comet logger for LLM
Cross-framework AI agent observability and reliability shim
Typed, versioned, regression-safe contracts for LLM calls
LLM Processing test
Detect silent incoherence in AI chain-of-thought reasoning
Configurable static and interactive visualizations for CrewAI Flows
A library for monitoring CrewAI agents, including token usage, cost calculation, logging, carbon emissions, and more.
Langfuse tracing integration for CrewAI workflows
Primordia economic metering for CrewAI - track crew costs with machine-readable receipts
CrewAI fork with per-agent and per-task token tracking. Track token usage for each agent and task to optimize costs and performance.
A comprehensive framework for evaluating Large Language Models with built-in support for bias, toxicity, relevancy metrics, custom evaluations, conversational test cases, release tracking, and token counting
Allow users to calculate RAG metrics
Accounting for LLM token usage
Track usage of LLM tokens in a SQLite table
The SDK client for communicating with Deepchecks LLM service
An integration of DeepEval LLM evaluation framework with Haystack
Deterministic-mode checks for LLM inference: measure run/batch variance, generate repro packs, and explain why outputs differ.
Advanced LLM token analysis and statistics toolkit for various data formats
DevSkin Monitor Agent - Python Instrumentation SDK
diRAGnosis - Diagnose the performance of your RAG!
Scan any codebase, see which files are burning your Claude/GPT tokens, and the exact dollar cost — with a CI budget gate and drop-in adapters for every major LLM.
Generates evaluation dataset for fine-tuning LLM
Microsoft AutoGen (v0.4+) integration for the Dominion Observatory — instrument autogen_core.tools.BaseTool calls with anonymised runtime telemetry to the cross-ecosystem MCP trust layer.
CrewAI event listener for the Dominion Observatory — report MCP tool behaviour from CrewAI crews and flows to the cross-ecosystem runtime trust layer.
LangChain / LangGraph integration for the Dominion Observatory — auto-report MCP tool calls and pre-flight trust-score gate any LangChain tool.
LlamaIndex integration for the Dominion Observatory — instrument llama_index.tools.mcp.McpToolSpec tool calls with anonymised runtime telemetry to the cross-ecosystem MCP trust layer.
Evaluation method for the DRAGON benchmark
Behavioral drift detection for LLM agents
Comprehensive logging and visualization infrastructure for DSPy GEPA optimization runs
DSPy program/pipeline inspector widget for Jupyter/VSCode Notebooks.
A DSPy metric function learning package
Lifecycle instrumentation for DSPy's RLM (Recursive Language Model).
Python storage API for the Dyff AI auditing platform.
EasyObs SDK: lightweight LLM/agent observability instrumentation via OpenTelemetry
TensorRT Edge-LLM Benchmark CLI Tool
A Comprehensive Benchmark for Large Language Model Efficiency
A python implementation of an Elastic Agent Client
OpenAI Instrumentation Package
llmo instrumentation for OpenTelemetry
Validation tool to compare a generated context by sLLM to reference context
A comprehensive evaluation toolkit for assessing Retrieval-Augmented Generation (RAG) outputs using linguistic, semantic, and fairness metrics
Simple LLM/RAG evaluation framework for teams
A flexible task evaluation client
QA framework for evaluating LLM outputs based on user-defined metrics
ClawBench: Can AI Agents Complete Everyday Online Tasks? (alias of claw-bench)
Lightweight reporting SDK for evo experiments. Zero dependencies.
Expect scripts for LLM conversations
Experiment tracker made for LLM interaction.
Local AI debugging agent - detect, group, score, debug, and explain API failures
Python implementation of dragonfly DB API, can be used for testing purposes.
A mock LLM server for testing. Drop-in replacement for OpenAI and Anthropic APIs.
Deterministic semantic assertions for LLM tests. Freeze a judge's verdict once; replay it forever.
Inspect and manage fakellm-assert frozen judgment snapshots from the terminal.
Record real OpenAI/Anthropic API traffic and emit fakellm.yaml rules automatically.
AI Agent Observability SDK for Python
LangGraph testing plugin for fasteval -- wrap any compiled StateGraph with a minimal harness for chat, session, mocking, and node-level testing
Fiddler SDK for instrumenting LangChain V1 agents with OpenTelemetry
LangGraph and LangChain instrumentation for Fiddler GenAI observability
金融AI Agent评测系统 - 标准化的评测框架
111 labeled eval cases, deterministic checks, and scoring for financial trading agents
Run different validation tests on machine learning models.
Freeplay integration for LangGraph and LangChain
Freeplay integration for LangGraph and LangChain
Get insights on your agent and app interations.
Local gateway for collecting and managing agent events with Google Sheets integration
Qualitatively evaluate LLMs
A Gym for Generalist LLMs.
Gemini CLI model quota monitor
Instrumentation and telemetry for the Google Gemini Live API via OpenTelemetry
Drop-in observability for Gemini agents: traces, cost, drift, egress allowlist. Ships exporters for Arize Phoenix, Splunk HEC, Elastic, GitLab Observability, MongoDB Atlas, Dynatrace, TrueFoundry.
Ghost Doc black-box tracing agent for Python
A library for evaluating LLM-based applications.
Goderash adapter for the Anthropic Python SDK (Messages + tool use) — tamper-evident, hash-chained, regulator-ready.
LangGraph / LangChain callback that emits Goderash audit events — tamper-evident, hash-chained, regulator-ready.
Goderash adapter for the OpenAI Python SDK (Chat + Responses + Assistants) — tamper-evident, hash-chained, regulator-ready.
A cost estimator for OpenAI API calls in tqdm loops.
Token usage and cost tracking for OpenAI responses (another AI providers in future)
Conduct surveys using GPT
Generate multiple draft responses and then use a second model to judge the answers and pick a winner, which is then returned to the user.
Probing chatgpt
A package to count tokens in input text using OpenAI's tiktoken library.
Agent Inspector for ADK
Library to monitor your OpenAI usage and send metrics and logs to Grafana Cloud
For assessing question answering systems' final answers and intermediate steps, against a given set of questions, reference answers and steps.
graphrag-with-trace
A package to evaluate output relevancy using Groq LLM
groqeval is a simple-to-use, open-source LLM evaluation framework that uses groq for evaluation.
Guidance platform for deploying and managing large language models.
Collection of Harbor agents for AI evaluation
Evaluation-First Control Layer for Enterprise RAG Systems