Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 39 of 57

llmvision

Visualize how LLMs tokenize text

llmwatch

Track, tag, and report LLM API costs by feature, user, and model.

lmnr-claude-code-proxy

Thin proxy server for Claude Code and Laminar tracing

local-agent-cost-tracker

Real-time AI agent cost monitoring and budget alerting for local development

loongsuite-instrumentation-agentscope

LoongSuite AgentScope Instrumentation

loongsuite-instrumentation-claude-agent-sdk

LoongSuite Claude Agent SDK instrumentation

loongsuite-instrumentation-crewai

LoongSuite CrewAI instrumentation

loongsuite-instrumentation-hermes-agent

LoongSuite Hermes Agent instrumentation

loongsuite-instrumentation-langchain

LoongSuite LangChain Instrumentation

loongsuite-instrumentation-langgraph

LoongSuite LangGraph Instrumentation

loongsuite-instrumentation-litellm

LoongSuite LiteLLM instrumentation

loongsuite-instrumentation-qwen-agent

LoongSuite Qwen-Agent Instrumentation

magenta-telemetry

Ultra-lightweight binary telemetry serialization format

magentic-olly

A package for custom opentelemetry instrumentation

magnific-llm-evals

A package for evaluating LLMs in customer service scenarios

mamood-llm-cost-estimator

Realtime OpenRouter LLM API call cost estimator

manowar-agent

Package to Add as a Collector

maos-agent

The Observability & Resilience SDK for Maos AI Agents

mate-engine-ollama

Ollama engine plugin for mate-bench

mate-workload-llm

LLM workload plugin for mate-bench

mce-ragas-adapter

RAGAS integration adapter for Metrics Computation Engine

mcp-llm-behave

MCP server exposing llm-behave behavioral regression testing as Claude tools

mcp-llm-eval

MCP server that packages LLM evaluation gates as reusable CI/CD primitives

meshagent-otel

CLI for Meshagent

meta-agents-research-environments

Meta Agents Research Environments is a research-driven environment designed to simulate complex, real-life tasks that span several minutes and require multiple steps to be solved. Unlike static simulation environments, this platform introduces a dynamic setting where the state of the environment evolves and new information is continuously integrated.

metricllm

Add your description here

metrist-agent-canmonuser

Metrist In-Process Agent

microsoft-agents-a365-observability-core

Telemetry, tracing, and monitoring components for AI agents

microsoft-agents-a365-observability-extensions-agent-framework

Agent Framework observability and tracing extensions for Microsoft Agent 365

microsoft-agents-a365-observability-extensions-langchain

LangChain observability and tracing extensions for Microsoft Agent 365

microsoft-agents-a365-observability-extensions-openai

OpenAI Agents SDK observability and tracing extensions for Microsoft Agent 365

microsoft-agents-a365-observability-extensions-semantic-kernel

Semantic Kernel observability and tracing extensions for Microsoft Agent 365

microsoft-agents-a365-observability-hosting

Hosting components for Agent 365 observability

microsoft-agents-a365-runtime

Telemetry, tracing, and monitoring components for AI agents

ming-probe-langchain

乾坤镜 LangChain 自动探针 —— pip install 即插即用,零代码侵入

mirage-bench

A RAG-based benchmark for multilingual question answering.

mirage-benchmark

A Multiagent Framework for Generating Multimodal Multihop QA Datasets for RAG Evaluation

mk-agentkit

The agent reliability stack in one install: agentfit + agentguard + agentsnap + agentvet + agentcast (Python ports).

mlflow-txtai

MLflow automatic tracing for txtai

mlhook-agent

mlhook-agent - temporary placeholder

mllm-shap

Shapley-value attributions for conversational language models.

mlrb-agent-tasks

A task package for ML Research Bench

mm-ragbench

Multimodal RAG benchmark dataset — companion to mmeval-vrag

mmeval-vrag

Evaluation framework for Multimodal Vision-Language RAG systems — measure retrieval quality, hallucination, faithfulness, and cross-modal alignment in one unified pipeline.

mockagent

A Python package for mocking agents

mockllm

A mock server that mimics OpenAI and Anthropic API formats for testing

moda-anthropic

Moda Anthropic instrumentation for LLM observability

moda-claude-agent-sdk

Moda Claude Agent SDK instrumentation for LLM observability

moda-openai

Moda OpenAI instrumentation for LLM observability

mona-openai

Integration client for monitoring OpenAI usage with Mona

monitor-claude

Web-based usage monitoring tool for Claude Code

monitoring-rag

A comprehensive, framework-agnostic library for evaluating Retrieval-Augmented Generation (RAG) pipelines.

mseep-agentic-radar

A python library for generating reports for agentic systems.

mseep-agentops

Observability and DevTool Platform for AI Agents

multiagent-eval

Open Source Multi-Agent System Evaluation Framework

multiagent-inspect

Multi-agent system for AI evaluations in AISI's inspect-ai framework

multiagent-trace-middleware

Observability middleware for multi-agent LangGraph/LangChain workflows. Track agent trees, emit events, and trace execution across nested agents.

multiagent-verify

Enterprise-grade reliability framework for AI agents.

murmur-langgraph

LangGraph client for murmur

mz-logging-azure-tablestorage

A professional logging module using Azure Table Storage

nail-agent

ClawBench: Can AI Agents Complete Everyday Online Tasks? (alias of claw-bench)

needlehaystack

Pressure-test LLM long-context retrieval with the Needle In A Haystack benchmark.

newrelic-plugins-agent

Python based agent for collecting metrics for NewRelic

newrelic-python-agent

Python based agent for collecting metrics for NewRelic

noos-langchain

LangChain callback handler for Noos — pre-delivery halt/warn decisions from an agent's LLM+tool event stream

notilens-langchain

NotiLens callback handler for LangChain — automatic run tracking, token metrics, and error alerting

nr-openai-observability

A lightweight tool to monitor your OpenAI workload.

nuaudit-python-autogen

Nuaudit

nvidia-nat-ragaai

Subpackage for RagaAI Catalyst integration in NeMo Agent Toolkit

nvidia-nat-ragas

Subpackage for RAGAS evaluators in NVIDIA NeMo Agent Toolkit

observability-agent

Opentrons observability relay agent

observable-agent

Unopinionated contract-based verification for AI agents

observeLLM

A python package for observing traces of your LLM application.

odigos-opentelemetry-instrumentation-openai-v2

OpenTelemetry Official OpenAI instrumentation

oh-my-agentic-score

Measure agentic coding performance using Thread-Based Engineering framework

ollama-benchmarker

Comprehensive benchmark suite for Ollama models: LLM, embedding, and reranking evaluation

ollama-profiler

CLI + TUI tool for benchmarking Ollama models side-by-side

ollama-spark

Terminal toolkit for local Ollama model recommendation, benchmarking, and comparison.

ollamabench

A benchmark tool for consumer hardware for AI inference using llm models, with version and quantization detection.

omnipulse-agent

The Agentic Control Plane for the OmniPulse Zero-Copy Hashing Architecture

oneagent-sdk

Dynatrace OneAgent SDK for Python

open-agent-observatory

The OpenTelemetry for AI agents — structured traces, semantic diffs, and failure pattern mining

open-agent-traces

Generate realistic multi-agent workflow traces with LLM-enriched content, semantic validation, and PM4Py compatibility

open-llm-benchmark

Evaluate the capability of open-source LLMs in Agent, formatted output, instruction following, long context retrieval, multilingual, coding, math and custom task.

openai-agents-opentelemetry

OpenTelemetry tracing processor for the OpenAI Agents SDK

openai-billing

Fetch openai billing info from command line

openai-cost-calculator

Exact, auditable LLM API cost calculation across 13 providers, with prices synced from official sources.

openai-cost-calculator-with-lanchain

A library to estimate OpenAI API costs based on token usage, integrating with Langchain. This is a fork from v1.1.1 of openai_cost_calculator by Orkun Kınay and Murat Barkın Kınay (https://github.com/orkunkinay/openai_cost_calculator).

openai-cost-logger

OpenAI Cost Logger

openai-cost-tracker
openai-function-tokens

A package to estimate token counts for messages AND functions in openai's chat completion API.

openai-image-token-counter

Calculate token costs for images in OpenAI Vision API

openai-messages-token-helper

A helper library for estimating tokens used by messages sent through OpenAI Chat Completions API.

openai-otel

openai-otel is a library that automatically instrument your OpenAI python code using OpenTelemetry

openai-pricing

Easily check the current costs of OpenAI's GPT and ChatGPT models.

openai-pricing-api

Simple and clean API for OpenAI model cost calculation and tracking with automatic pricing updates

openai-pricing-calc-draft

LLM Cost Calculation

openai-pricing-logger

A package to log OpenAI API costs

openai-pricing-scraper

Simple way to fetch and parse the pricing data from the OpenAI pricing page.

openai-python-mocks

More in Agent frameworks & infrastructure