Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 34 of 57

cohorte-ragas

Pluggable agent backends for RAGAS dataset generation (default Llama Stack Responses API or LangGraph)

colber-autogen

AutoGen 0.4+ integration for the Colber platform — per-tool observability, AutoGen Memory adapter, and a 5-service toolkit of typed BaseTool[Args, Result] subclasses.

colber-crewai

CrewAI integration for the Colber platform — step/task callbacks, long-term memory, and a 5-service toolkit.

colber-langchain

LangChain integration for the Colber platform — callbacks, memory, and a 6-service toolkit.

comet-llm

Comet logger for LLM

conduit-ai-agents

Cross-framework AI agent observability and reliability shim

contractllm

Typed, versioned, regression-safe contracts for LLM calls

cookieLLMTest

LLM Processing test

cot-coherence

Detect silent incoherence in AI chain-of-thought reasoning

crewai-flowviz

Configurable static and interactive visualizations for CrewAI Flows

crewai-logging-patch
crewai-monitoring

A library for monitoring CrewAI agents, including token usage, cost calculation, logging, carbon emissions, and more.

crewai-observe

Langfuse tracing integration for CrewAI workflows

crewai-primordia

Primordia economic metering for CrewAI - track crew costs with machine-readable receipts

crewai-token-tracking

CrewAI fork with per-agent and per-task token tracking. Track token usage for each agent and task to optimize costs and performance.

cria-plugin-pack-rag-evaluator
custom-llm-eval

A comprehensive framework for evaluating Large Language Models with built-in support for bias, toxicity, relevancy metrics, custom evaluations, conversational test cases, release tracking, and token counting

customgpt-metrics

Allow users to calculate RAG metrics

datasette-llm-accountant

Accounting for LLM token usage

datasette-llm-usage

Track usage of LLM tokens in a SQLite table

deepchecks-llm-client

The SDK client for communicating with Deepchecks LLM service

deepeval-haystack

An integration of DeepEval LLM evaluation framework with Haystack

detllm

Deterministic-mode checks for LLM inference: measure run/batch variance, generate repro packs, and explain why outputs differ.

deus-llm-token-stats-guru

Advanced LLM token analysis and statistics toolkit for various data formats

devskin-agent

DevSkin Monitor Agent - Python Instrumentation SDK

diragnosis

diRAGnosis - Diagnose the performance of your RAG!

distill-llm

Scan any codebase, see which files are burning your Claude/GPT tokens, and the exact dollar cost — with a CI budget gate and drop-in adapters for every major LLM.

docsrag

Generates evaluation dataset for fine-tuning LLM

dominion-observatory-autogen

Microsoft AutoGen (v0.4+) integration for the Dominion Observatory — instrument autogen_core.tools.BaseTool calls with anonymised runtime telemetry to the cross-ecosystem MCP trust layer.

dominion-observatory-crewai

CrewAI event listener for the Dominion Observatory — report MCP tool behaviour from CrewAI crews and flows to the cross-ecosystem runtime trust layer.

dominion-observatory-langchain

LangChain / LangGraph integration for the Dominion Observatory — auto-report MCP tool calls and pre-flight trust-score gate any LangChain tool.

dominion-observatory-llamaindex

LlamaIndex integration for the Dominion Observatory — instrument llama_index.tools.mcp.McpToolSpec tool calls with anonymised runtime telemetry to the cross-ecosystem MCP trust layer.

dragon-eval

Evaluation method for the DRAGON benchmark

drift-detector-agent

Behavioral drift detection for LLM agents

dspy-gepa-logger

Comprehensive logging and visualization infrastructure for DSPy GEPA optimization runs

dspy-inspector

DSPy program/pipeline inspector widget for Jupyter/VSCode Notebooks.

dspy-metric-learning

A DSPy metric function learning package

dspy-rlm-hooks

Lifecycle instrumentation for DSPy's RLM (Recursive Language Model).

dyff-storage

Python storage API for the Dyff AI auditing platform.

eagent-eval
easyobs-agent

EasyObs SDK: lightweight LLM/agent observability instrumentation via OpenTelemetry

edgellm-bench

TensorRT Edge-LLM Benchmark CLI Tool

efficientllm-toolkit

A Comprehensive Benchmark for Large Language Model Efficiency

elastic-agent-client

A python implementation of an Elastic Agent Client

elastic-opentelemetry-instrumentation-openai

OpenAI Instrumentation Package

enova-instrumentation-llmo

llmo instrumentation for OpenTelemetry

equivalent_llm

Validation tool to compare a generated context by sLLM to reference context

eval-agent
eval-rag

A comprehensive evaluation toolkit for assessing Retrieval-Augmented Generation (RAG) outputs using linguistic, semantic, and fairness metrics

evalkit-llm

Simple LLM/RAG evaluation framework for teams

evallm

A flexible task evaluation client

evallm-qa

QA framework for evaluating LLM outputs based on user-defined metrics

everyday-agent

ClawBench: Can AI Agents Complete Everyday Online Tasks? (alias of claw-bench)

evo-hq-agent

Lightweight reporting SDK for evo experiments. Zero dependencies.

expectllm

Expect scripts for LLM conversations

exprag

Experiment tracker made for LLM interaction.

faillens-agent

Local AI debugging agent - detect, group, score, debug, and explain API failures

fake-dragonflydb

Python implementation of dragonfly DB API, can be used for testing purposes.

fakellm

A mock LLM server for testing. Drop-in replacement for OpenAI and Anthropic APIs.

fakellm-assert

Deterministic semantic assertions for LLM tests. Freeze a judge's verdict once; replay it forever.

fakellm-cli

Inspect and manage fakellm-assert frozen judgment snapshots from the terminal.

fakellm-recorder

Record real OpenAI/Anthropic API traffic and emit fakellm.yaml rules automatically.

farzanhossans-agentlens

AI Agent Observability SDK for Python

fasteval-langgraph

LangGraph testing plugin for fasteval -- wrap any compiled StateGraph with a minimal harness for chat, session, mocking, and node-level testing

fiddler-langchain

Fiddler SDK for instrumenting LangChain V1 agents with OpenTelemetry

fiddler-langgraph

LangGraph and LangChain instrumentation for Fiddler GenAI observability

finagent-eval

金融AI Agent评测系统 - 标准化的评测框架

finagent-evals

111 labeled eval cases, deterministic checks, and scoring for financial trading agents

fluidgpt

Run different validation tests on machine learning models.

freeplay-langgraph

Freeplay integration for LangGraph and LangChain

freeplay-python-langgraph

Freeplay integration for LangGraph and LangChain

gateagent

Get insights on your agent and app interations.

gateagent-gateway

Local gateway for collecting and managing agent events with Google Sheets integration

gauge-llm

Qualitatively evaluate LLMs

gem-llm

A Gym for Generalist LLMs.

gemini-cli-usage

Gemini CLI model quota monitor

gemini-live-telemetry

Instrumentation and telemetry for the Google Gemini Live API via OpenTelemetry

geminilens

Drop-in observability for Gemini agents: traces, cost, drift, egress allowlist. Ships exporters for Arize Phoenix, Splunk HEC, Elastic, GitLab Observability, MongoDB Atlas, Dynatrace, TrueFoundry.

ghost-doc-agent

Ghost Doc black-box tracing agent for Python

gllm-evals-binary

A library for evaluating LLM-based applications.

goderash-adapter-anthropic

Goderash adapter for the Anthropic Python SDK (Messages + tool use) — tamper-evident, hash-chained, regulator-ready.

goderash-adapter-langgraph

LangGraph / LangChain callback that emits Goderash audit events — tamper-evident, hash-chained, regulator-ready.

goderash-adapter-openai

Goderash adapter for the OpenAI Python SDK (Chat + Responses + Assistants) — tamper-evident, hash-chained, regulator-ready.

gpt-cost-estimator

A cost estimator for OpenAI API calls in tqdm loops.

gpt-token-tracker

Token usage and cost tracking for OpenAI responses (another AI providers in future)

gptbatcher

Conduct surveys using GPT

gptgladiator

Generate multiple draft responses and then use a second model to judge the answers and pick a winner, which is then returned to the user.

gptprobe

Probing chatgpt

gptwc

A package to count tokens in input text using OpenAI's tiktoken library.

gradio-agent-inspector

Agent Inspector for ADK

grafana-openai-monitoring

Library to monitor your OpenAI usage and send metrics and logs to Grafana Cloud

graphrag-eval

For assessing question answering systems' final answers and intermediate steps, against a given set of questions, reference answers and steps.

graphrag-with-trace

graphrag-with-trace

groq-eval-score

A package to evaluate output relevancy using Groq LLM

groqeval

groqeval is a simple-to-use, open-source LLM evaluation framework that uses groq for evaluation.

gstorage-backup
gstorage-backupx
guidellm

Guidance platform for deploying and managing large language models.

harbor-agents

Collection of Harbor agents for AI evaluation

hardrag-core

Evaluation-First Control Layer for Enterprise RAG Systems

More in Agent frameworks & infrastructure