Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 35 of 57

haystack-client

Haystack Python OpenTracing Implementation

haystackbench

A comprehensive, flexible NIAH evaluation framework for LLMs

hellmholtz

A comprehensive Python package for unified LLM access, benchmarking, evaluation, and reporting

hiagent-observe

HiAgent Observe SDK

hiveary-agent

Hiveary Monitoring Agent

honeagents-hone

Hone SDK - AI Experience Engineering Platform for tracking and improving LLM applications

hookedllm

Async-first, scoped hook system for LLM observability

hydrag-benchmark

Local-only RAG benchmarking CLI — measures recall, MRR, chunk overlap, latency, and BEIR IR metrics

hyperstack-langgraph

The Agent Provenance Graph for LangGraph agents. Timestamped facts, auditable decisions, deterministic trust. Prove what agents knew, trace why they knew it, coordinate without an LLM in the loop. $0 per operation.

hypertrace-agent

Hypertrace Python Agent

ibm-agent-analytics

Agent Analytics observability SDK package

ibm-agent-analytics-common

Agent Analytics observability Common package

ibm-agent-analytics-core

Agent Analytics Core observability SDK package

identity-trace-python-agent

Tracing agent for python.

im-openai

Wrapper library for openai to send events to the Imaginary Programming monitor

industrial-mllm-benchmark

Code to benchmark and evaluate MLLM on industrial data (data itself are not part of the publication).

infinium-claude-code-connector

Bridge between Claude Code hooks and Infinium agent tracing

info-llm-observe

A small observability wrapper for LLM projects (Phoenix adapter included)

infosrv-llm-observe

A small observability wrapper for LLM projects (Phoenix adapter included)

insidellm

Python SDK for LLM/Agent Analytics Platform

isage-agentic-tooluse-benchmark

SAGE Tool Use Benchmark - Tool selection and use evaluation framework

isage-agentic-tooluse-sias

SAGE Tool Use SIAS - Sample-Importance-Aware Selection for tool selection and agent training

isage-benchmark-agent

SAGE Benchmark Agent - Agent capability evaluation framework

isagellm-benchmark

Benchmark Suite & E2E Testing for sageLLM

isagellm-control-plane-benchmark

Control Plane scheduling benchmark for sageLLM (formerly isage-control-plane-benchmark)

japanese_needlehaystack

https://github.com/gkamradt/LLMTest_NeedleInAHaystack を元にいくつか修正を加えたものです。

judge-llm

A lightweight LLM evaluation framework for comparing and testing AI providers

jumper-logging-agent

The jumper background logging agent for Linux and Windows systems

k-agentops

K-AgentOps SDK - Agent observability, tracing, and evaluation metrics

kaizen-agent

An AI debugging engineer that continuously tests, analyzes, and improves your AI agents and LLM applications

kantan-agents

kantan-agents is a thin, opinionated wrapper around the OpenAI Agents SDK that makes observability and evaluation “just happen” by default.

keel-llm-otel

OpenTelemetry metrics + span events for keel-llm-reliability. A pure wrapper around ResilientClient — emits per-attempt metrics and span events from the visible-degradation trail. No coupling: keel-llm-reliability stays observability-free.

keywordsai-exporter-haystack

Keywords AI integration for Haystack pipelines with tracing and logging support

keywordsai-exporter-openai-agents

Exporter for Keywords AI openai Agents SDK

klg-llm-audit

Internal LLM audit logger for maybeai services

ko-llm-eval

Korean LLM evaluation orchestration layer with multi-judge scoring.

kpi-agentes

SDK para monitoramento e observabilidade de agentes de IA — KPI Agentes

krag

A Python package for RAG performance evaluation

ktds-agent-ops-sdk

KT DS AgentOps SDK - Agent observability, tracing, and evaluation metrics

langchain-base-tests

Standard tests for LangChain implementations

langchain-benchmarks

🦜💪 Flex those feathers!

langchain-callback-parquet-logger

A Parquet-based callback handler for logging LangChain LLM interactions

langchain-chaos-middleware

Chaos testing middleware for LangChain agents

langchain-critique

An integration package connecting Critique and LangChain

langchain-data-validator

LangChain tool for data validation and fact-checking. Helps AI agents avoid hallucinations.

langchain-goodfire

Goodfire integration for LangChain

langchain-instrumentation

OpenTelemetry instrumentation for LangChain and LangGraph spans and metrics.

langchain-integration-health

Comprehensive testing framework and dashboard for LangChain integration compatibility

langchain-llm-streamer

Stream and print responses from LangChain's language models asynchronously.

langchain-logger

A Chain of Thought logger for langchain

langchain-longprobe

LangChain integration for LongProbe — sub-second RAG retrieval regression testing with chunk-level diffing

langchain-oso

Oso observability integration for LangChain agents

langchain-plantuml

Subscribe to events using a callback and store them in PlantUML format. You can easily subscribe to events and keep them in a form that is easy to visualize and analyze.

langchain-prefect

Orchestrate and observe tools built with langchain.

langchain-primordia

Primordia economic metering for LangChain - track AI costs with machine-readable receipts

langchain-progress

Wrapper for nicely displaying progress bars for langchain embedding components when using multiprocessing or ray.

langchain-qaip

An integration package connecting QAIP and LangChain

langchain-replay

Record and replay LangChain/LangGraph agent decisions while tools execute against the real filesystem.

langchain-tests

Standard tests for LangChain implementations

langchain-token-usage

Langchain Token Usage

langchain-visualizer

Visualization and debugging tool for LangChain workflows

langchain-widget

A widget to run in Jupyter Notebook.

langchainplus-sdk

Client library to connect to the LangSmith LLM Tracing and Evaluation Platform.

langevals-haystack

LangEvals Haystack evaluator

langevals-ragas

LangEvals Ragas evaluator

langfuse-agent

Agent for interacting with Langfuse Observability API

langfuse-haystack

Langfuse integration for Haystack

langfuse-instrumented-dspy-lm

Custom DSPy LM class for token usage tracking with Langfuse

langfuse-langchain

A client library for accessing langfuse

langfuse-lens-agent

Langfuse trace analysis agent with built-in web console

langfuse-pydantic-ai

langfuse-pydantic-ai

langgraph-chat-agent-eval

Evaluation framework for LangGraph chatbots on multi-turn conversations

langgraph-checkpoint-conformance

Conformance test suite for LangGraph checkpointer implementations.

langgraph-compare

Package for comparing and monitoring performance of LangGraph architectures.

langgraph-lens

Zero-config runtime observability for LangGraph agents: checkpoint, prompt-supply-chain, tool, memory, PII, goal-hijack, inter-agent, and SQL-injection detectors emitted as structured events.

langgraph-node-streamer

Node-level streaming for LangGraph graphs

langgraph-stream-parser

Renamed to langstage-core. This is a thin compatibility shim that re-exports langstage-core under the old import name.

langgraph-tracer

A small package for langgraph trace

langgraph-viz

Real-time browser-based visualization for LangGraph workflows

langgraphics

Debug LangGraph-based agents visually - live execution graph, node inspection, and run replay, all from a single line.

langinsight-langchain

LangInsight callback handler for LangChain - Monitor and trace your LangChain LLM applications

lantern-llm

AI observability and security for African tech companies

lemma-agent

The GitHub Copilot for AI Agent Development - Intelligent debugging for AI agents

libretto-openai

Wrapper library for openai to send events to Libretto

lifesaver-lite-llm

Lightweight LLM usage analysis and dashboard with mitmweb ingestion

lighthousellmm

Client library to connect to the lighthousellmm LLM Tracing and Evaluation Platform.

litellm-cost-tracker

A lightweight Python library for tracking LLM API costs via litellm's callback system

litellm-impacts

Environmental impact metrics callback for LiteLLM

litmus-llm

Litmus — LLM scenario runner TUI

llama

LLAMA - Loss & LAtency MAtrix

llama-api-bench

Llama API Benchmark

llama-benchy

llama-bench style benchmarking tool for all OpenAI-compatible LLM endpoints

llama-cloud-fake

A respx-backed fake of the LlamaCloud v2 API for use in tests

llama-dashboard

Web dashboard for monitoring and interacting with the LlamaAI Ecosystem.

llama-index-callbacks-agentops

llama-index instrumentation agentops integration

llama-index-callbacks-aim

llama-index callbacks aim integration

llama-index-callbacks-argilla

llama-index callbacks argilla integration

llama-index-callbacks-arize-phoenix

llama-index callbacks arize-phoenix integration

llama-index-callbacks-deepeval

llama-index callbacks deepeval integration

llama-index-callbacks-honeyhive

llama-index callbacks honeyhive integration

More in Agent frameworks & infrastructure