Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 39 of 57
Visualize how LLMs tokenize text
Track, tag, and report LLM API costs by feature, user, and model.
Thin proxy server for Claude Code and Laminar tracing
Real-time AI agent cost monitoring and budget alerting for local development
LoongSuite AgentScope Instrumentation
LoongSuite Claude Agent SDK instrumentation
LoongSuite CrewAI instrumentation
LoongSuite Hermes Agent instrumentation
LoongSuite LangChain Instrumentation
LoongSuite LangGraph Instrumentation
LoongSuite LiteLLM instrumentation
LoongSuite Qwen-Agent Instrumentation
Ultra-lightweight binary telemetry serialization format
A package for custom opentelemetry instrumentation
A package for evaluating LLMs in customer service scenarios
Realtime OpenRouter LLM API call cost estimator
Package to Add as a Collector
The Observability & Resilience SDK for Maos AI Agents
Ollama engine plugin for mate-bench
LLM workload plugin for mate-bench
RAGAS integration adapter for Metrics Computation Engine
MCP server exposing llm-behave behavioral regression testing as Claude tools
MCP server that packages LLM evaluation gates as reusable CI/CD primitives
CLI for Meshagent
Meta Agents Research Environments is a research-driven environment designed to simulate complex, real-life tasks that span several minutes and require multiple steps to be solved. Unlike static simulation environments, this platform introduces a dynamic setting where the state of the environment evolves and new information is continuously integrated.
Add your description here
Metrist In-Process Agent
Telemetry, tracing, and monitoring components for AI agents
Agent Framework observability and tracing extensions for Microsoft Agent 365
LangChain observability and tracing extensions for Microsoft Agent 365
OpenAI Agents SDK observability and tracing extensions for Microsoft Agent 365
Semantic Kernel observability and tracing extensions for Microsoft Agent 365
Hosting components for Agent 365 observability
Telemetry, tracing, and monitoring components for AI agents
乾坤镜 LangChain 自动探针 —— pip install 即插即用,零代码侵入
A RAG-based benchmark for multilingual question answering.
A Multiagent Framework for Generating Multimodal Multihop QA Datasets for RAG Evaluation
The agent reliability stack in one install: agentfit + agentguard + agentsnap + agentvet + agentcast (Python ports).
MLflow automatic tracing for txtai
mlhook-agent - temporary placeholder
Shapley-value attributions for conversational language models.
A task package for ML Research Bench
Multimodal RAG benchmark dataset — companion to mmeval-vrag
Evaluation framework for Multimodal Vision-Language RAG systems — measure retrieval quality, hallucination, faithfulness, and cross-modal alignment in one unified pipeline.
A Python package for mocking agents
A mock server that mimics OpenAI and Anthropic API formats for testing
Moda Anthropic instrumentation for LLM observability
Moda Claude Agent SDK instrumentation for LLM observability
Moda OpenAI instrumentation for LLM observability
Integration client for monitoring OpenAI usage with Mona
Web-based usage monitoring tool for Claude Code
A comprehensive, framework-agnostic library for evaluating Retrieval-Augmented Generation (RAG) pipelines.
A python library for generating reports for agentic systems.
Observability and DevTool Platform for AI Agents
Open Source Multi-Agent System Evaluation Framework
Multi-agent system for AI evaluations in AISI's inspect-ai framework
Observability middleware for multi-agent LangGraph/LangChain workflows. Track agent trees, emit events, and trace execution across nested agents.
Enterprise-grade reliability framework for AI agents.
LangGraph client for murmur
A professional logging module using Azure Table Storage
ClawBench: Can AI Agents Complete Everyday Online Tasks? (alias of claw-bench)
Pressure-test LLM long-context retrieval with the Needle In A Haystack benchmark.
Python based agent for collecting metrics for NewRelic
Python based agent for collecting metrics for NewRelic
LangChain callback handler for Noos — pre-delivery halt/warn decisions from an agent's LLM+tool event stream
NotiLens callback handler for LangChain — automatic run tracking, token metrics, and error alerting
A lightweight tool to monitor your OpenAI workload.
Nuaudit
Subpackage for RagaAI Catalyst integration in NeMo Agent Toolkit
Subpackage for RAGAS evaluators in NVIDIA NeMo Agent Toolkit
Opentrons observability relay agent
Unopinionated contract-based verification for AI agents
A python package for observing traces of your LLM application.
OpenTelemetry Official OpenAI instrumentation
Measure agentic coding performance using Thread-Based Engineering framework
Comprehensive benchmark suite for Ollama models: LLM, embedding, and reranking evaluation
CLI + TUI tool for benchmarking Ollama models side-by-side
Terminal toolkit for local Ollama model recommendation, benchmarking, and comparison.
A benchmark tool for consumer hardware for AI inference using llm models, with version and quantization detection.
The Agentic Control Plane for the OmniPulse Zero-Copy Hashing Architecture
Dynatrace OneAgent SDK for Python
The OpenTelemetry for AI agents — structured traces, semantic diffs, and failure pattern mining
Generate realistic multi-agent workflow traces with LLM-enriched content, semantic validation, and PM4Py compatibility
Evaluate the capability of open-source LLMs in Agent, formatted output, instruction following, long context retrieval, multilingual, coding, math and custom task.
OpenTelemetry tracing processor for the OpenAI Agents SDK
Fetch openai billing info from command line
Exact, auditable LLM API cost calculation across 13 providers, with prices synced from official sources.
A library to estimate OpenAI API costs based on token usage, integrating with Langchain. This is a fork from v1.1.1 of openai_cost_calculator by Orkun Kınay and Murat Barkın Kınay (https://github.com/orkunkinay/openai_cost_calculator).
OpenAI Cost Logger
A package to estimate token counts for messages AND functions in openai's chat completion API.
Calculate token costs for images in OpenAI Vision API
A helper library for estimating tokens used by messages sent through OpenAI Chat Completions API.
openai-otel is a library that automatically instrument your OpenAI python code using OpenTelemetry
Easily check the current costs of OpenAI's GPT and ChatGPT models.
Simple and clean API for OpenAI model cost calculation and tracking with automatic pricing updates
LLM Cost Calculation
A package to log OpenAI API costs
Simple way to fetch and parse the pricing data from the OpenAI pricing page.