Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 35 of 57
Haystack Python OpenTracing Implementation
A comprehensive, flexible NIAH evaluation framework for LLMs
A comprehensive Python package for unified LLM access, benchmarking, evaluation, and reporting
HiAgent Observe SDK
Hiveary Monitoring Agent
Hone SDK - AI Experience Engineering Platform for tracking and improving LLM applications
Async-first, scoped hook system for LLM observability
Local-only RAG benchmarking CLI — measures recall, MRR, chunk overlap, latency, and BEIR IR metrics
The Agent Provenance Graph for LangGraph agents. Timestamped facts, auditable decisions, deterministic trust. Prove what agents knew, trace why they knew it, coordinate without an LLM in the loop. $0 per operation.
Hypertrace Python Agent
Agent Analytics observability SDK package
Agent Analytics observability Common package
Agent Analytics Core observability SDK package
Tracing agent for python.
Wrapper library for openai to send events to the Imaginary Programming monitor
Code to benchmark and evaluate MLLM on industrial data (data itself are not part of the publication).
Bridge between Claude Code hooks and Infinium agent tracing
A small observability wrapper for LLM projects (Phoenix adapter included)
A small observability wrapper for LLM projects (Phoenix adapter included)
Python SDK for LLM/Agent Analytics Platform
SAGE Tool Use Benchmark - Tool selection and use evaluation framework
SAGE Tool Use SIAS - Sample-Importance-Aware Selection for tool selection and agent training
SAGE Benchmark Agent - Agent capability evaluation framework
Benchmark Suite & E2E Testing for sageLLM
Control Plane scheduling benchmark for sageLLM (formerly isage-control-plane-benchmark)
https://github.com/gkamradt/LLMTest_NeedleInAHaystack を元にいくつか修正を加えたものです。
A lightweight LLM evaluation framework for comparing and testing AI providers
The jumper background logging agent for Linux and Windows systems
K-AgentOps SDK - Agent observability, tracing, and evaluation metrics
An AI debugging engineer that continuously tests, analyzes, and improves your AI agents and LLM applications
kantan-agents is a thin, opinionated wrapper around the OpenAI Agents SDK that makes observability and evaluation “just happen” by default.
OpenTelemetry metrics + span events for keel-llm-reliability. A pure wrapper around ResilientClient — emits per-attempt metrics and span events from the visible-degradation trail. No coupling: keel-llm-reliability stays observability-free.
Keywords AI integration for Haystack pipelines with tracing and logging support
Exporter for Keywords AI openai Agents SDK
Internal LLM audit logger for maybeai services
Korean LLM evaluation orchestration layer with multi-judge scoring.
SDK para monitoramento e observabilidade de agentes de IA — KPI Agentes
A Python package for RAG performance evaluation
KT DS AgentOps SDK - Agent observability, tracing, and evaluation metrics
Standard tests for LangChain implementations
🦜💪 Flex those feathers!
A Parquet-based callback handler for logging LangChain LLM interactions
Chaos testing middleware for LangChain agents
An integration package connecting Critique and LangChain
LangChain tool for data validation and fact-checking. Helps AI agents avoid hallucinations.
Goodfire integration for LangChain
OpenTelemetry instrumentation for LangChain and LangGraph spans and metrics.
Comprehensive testing framework and dashboard for LangChain integration compatibility
Stream and print responses from LangChain's language models asynchronously.
A Chain of Thought logger for langchain
LangChain integration for LongProbe — sub-second RAG retrieval regression testing with chunk-level diffing
Oso observability integration for LangChain agents
Subscribe to events using a callback and store them in PlantUML format. You can easily subscribe to events and keep them in a form that is easy to visualize and analyze.
Orchestrate and observe tools built with langchain.
Primordia economic metering for LangChain - track AI costs with machine-readable receipts
Wrapper for nicely displaying progress bars for langchain embedding components when using multiprocessing or ray.
An integration package connecting QAIP and LangChain
Record and replay LangChain/LangGraph agent decisions while tools execute against the real filesystem.
Standard tests for LangChain implementations
Langchain Token Usage
Visualization and debugging tool for LangChain workflows
A widget to run in Jupyter Notebook.
Client library to connect to the LangSmith LLM Tracing and Evaluation Platform.
LangEvals Haystack evaluator
LangEvals Ragas evaluator
Agent for interacting with Langfuse Observability API
Langfuse integration for Haystack
Custom DSPy LM class for token usage tracking with Langfuse
A client library for accessing langfuse
Langfuse trace analysis agent with built-in web console
langfuse-pydantic-ai
Evaluation framework for LangGraph chatbots on multi-turn conversations
Conformance test suite for LangGraph checkpointer implementations.
Package for comparing and monitoring performance of LangGraph architectures.
Zero-config runtime observability for LangGraph agents: checkpoint, prompt-supply-chain, tool, memory, PII, goal-hijack, inter-agent, and SQL-injection detectors emitted as structured events.
Node-level streaming for LangGraph graphs
Renamed to langstage-core. This is a thin compatibility shim that re-exports langstage-core under the old import name.
A small package for langgraph trace
Real-time browser-based visualization for LangGraph workflows
Debug LangGraph-based agents visually - live execution graph, node inspection, and run replay, all from a single line.
LangInsight callback handler for LangChain - Monitor and trace your LangChain LLM applications
AI observability and security for African tech companies
The GitHub Copilot for AI Agent Development - Intelligent debugging for AI agents
Wrapper library for openai to send events to Libretto
Lightweight LLM usage analysis and dashboard with mitmweb ingestion
Client library to connect to the lighthousellmm LLM Tracing and Evaluation Platform.
A lightweight Python library for tracking LLM API costs via litellm's callback system
Environmental impact metrics callback for LiteLLM
Litmus — LLM scenario runner TUI
LLAMA - Loss & LAtency MAtrix
Llama API Benchmark
llama-bench style benchmarking tool for all OpenAI-compatible LLM endpoints
A respx-backed fake of the LlamaCloud v2 API for use in tests
Web dashboard for monitoring and interacting with the LlamaAI Ecosystem.
llama-index instrumentation agentops integration
llama-index callbacks aim integration
llama-index callbacks argilla integration
llama-index callbacks arize-phoenix integration
llama-index callbacks deepeval integration
llama-index callbacks honeyhive integration