Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 40 of 57

openai-request-tracker

OpenAI cost tracker that logs OpenAI requests using opentelemetry

openai-responses

🧪🤖 Pytest plugin for automatically mocking OpenAI requests

openai-tests

Quick CLI to test OpenAI-compatible API endpoints.

openai-token-counter

OpenAI Token Counter

openai-usage

Simple Library for OpenAI Usage

openai-usage-logger

A utility for logging OpenAI API usage and costs.

openai-usage-report

Console OpenAI API cost and token usage analysis by key and model

openai-vision-cost

Calculate costs for OpenAI vision model image inputs

openai-wrapper

A wrapper for OpenAI's python API which wraps around the openAI functions and stores the request, response and metadata to MongoDB. The stored data can be used to fine tune GPT-3 models or HuggingFace models.

openaibox

Universal LLM introspection. Understand any model. Any architecture.

openaiisrich

latest pricing information of openai models, crowd maintained

openapi-agent-benchmark

Add your description here

openbox-deepagent-sdk-python

OpenBox governance SDK for DeepAgents (langchain-ai/deepagents)

openbox-langchain-sdk-python

OpenBox governance and observability SDK for LangChain

openbox-langgraph-sdk-python

OpenBox governance and observability SDK for LangGraph

opencensus-ext-ocagent

OpenCensus OC-Agent Trace Exporter

openclaw-central-agent

Node agent for OpenClaw Central distributed monitoring platform

openinference-instrumentation-agent-framework

OpenInference span processor for Microsoft Agent Framework - transforms native OpenTelemetry spans to OpenInference format

openinference-instrumentation-agentminds

OpenInference auto-instrumentation for the AgentMinds Python SDK. Emits OTel spans tagged with openinference.span.kind + agentminds.* attributes when agentminds.capture_* and agentminds.init are called.

openinference-instrumentation-agentspec

OpenInference Open Agent Spec Instrumentation

openinference-instrumentation-anthropic

OpenInference Anthropic Instrumentation

openinference-instrumentation-autogen

Deprecated compatibility package for OpenInference AG2 instrumentation

openinference-instrumentation-autogen-agentchat

OpenInference Autogen-Agentchat Instrumentation

openinference-instrumentation-bedrock

OpenInference Bedrock Instrumentation

openinference-instrumentation-claude-agent-sdk

OpenInference Claude Agent SDK Instrumentation

openinference-instrumentation-crewai

OpenInference Crewai Instrumentation

openinference-instrumentation-dspy

OpenInference DSPy Instrumentation

openinference-instrumentation-groq

OpenInference Groq Instrumentation

openinference-instrumentation-haystack

OpenInference Haystack Instrumentation

openinference-instrumentation-langchain

OpenInference LangChain Instrumentation

openinference-instrumentation-litellm

OpenInference liteLLM Instrumentation

openinference-instrumentation-llama-index

OpenInference LlamaIndex Instrumentation

openinference-instrumentation-mistralai

OpenInference Mistral AI Instrumentation

openinference-instrumentation-monkai-agent

OpenInference instrumentation for MonkAI Agent

openinference-instrumentation-openai

OpenInference OpenAI Instrumentation

openinference-instrumentation-openai-agents

OpenInference OpenAI Agents Instrumentation

openinference-instrumentation-openllmetry

OpenInference OpenLLMetry Instrumentation

openinference-instrumentation-pydantic-ai

OpenInference PydanticAI Instrumentation

openinference-instrumentation-smolagents

OpenInference smolagents Instrumentation

openinference-instrumentation-strands-agents

OpenInference span processor for Strands Agents - transforms native OpenTelemetry spans to OpenInference format

openllmtrace

End-to-end observability with built-in security guardrails.

opentelemetry-claude-agent-sdk

OpenTelemetry instrumentation for the Anthropic Claude Agent SDK

opentelemetry-if-instrumentation-anthropic

OpenTelemetry Anthropic instrumentation

opentelemetry-if-instrumentation-bedrock

OpenTelemetry Bedrock instrumentation

opentelemetry-if-instrumentation-cohere

OpenTelemetry Cohere instrumentation

opentelemetry-if-instrumentation-haystack

OpenTelemetry Haystack instrumentation

opentelemetry-if-instrumentation-langchain

OpenTelemetry Langchain instrumentation

opentelemetry-if-instrumentation-llamaindex

OpenTelemetry LlamaIndex instrumentation

opentelemetry-if-instrumentation-mistralai

OpenTelemetry Mistral AI instrumentation

opentelemetry-if-instrumentation-ollama

OpenTelemetry Ollama instrumentation

opentelemetry-if-instrumentation-openai

OpenTelemetry OpenAI instrumentation

opentelemetry-instrument-anthropic

Instruments Anthropic client for OpenTelemetry

opentelemetry-instrument-openai

OpenTelemetry openai instrumentation

opentelemetry-instrumentation-anthropic

OpenTelemetry Anthropic instrumentation

opentelemetry-instrumentation-bedrock

OpenTelemetry Bedrock instrumentation

opentelemetry-instrumentation-bedrock-tk-karini

OpenTelemetry Bedrock instrumentation

opentelemetry-instrumentation-cohere

OpenTelemetry Cohere instrumentation

opentelemetry-instrumentation-crewai

OpenTelemetry crewAI instrumentation

opentelemetry-instrumentation-groq

OpenTelemetry Groq instrumentation

opentelemetry-instrumentation-haystack

OpenTelemetry Haystack instrumentation

opentelemetry-instrumentation-langchain

OpenTelemetry Langchain instrumentation

opentelemetry-instrumentation-langchain-openai-embeddings

OpenTelemetry langchain_openai.embeddings instrumentation

opentelemetry-instrumentation-langchain-tk-karini

OpenTelemetry Langchain instrumentation

opentelemetry-instrumentation-llamaindex

OpenTelemetry LlamaIndex instrumentation

opentelemetry-instrumentation-mistralai

OpenTelemetry Mistral AI instrumentation

opentelemetry-instrumentation-ollama

OpenTelemetry Ollama instrumentation

opentelemetry-instrumentation-openai

OpenTelemetry OpenAI instrumentation

opentelemetry-instrumentation-openai-agents

OpenTelemetry OpenAI Agents instrumentation

opentelemetry-instrumentation-openai-agents-v2

OpenTelemetry OpenAI Agents instrumentation (barebones)

opentelemetry-instrumentation-openai-tk-karini

OpenTelemetry OpenAI instrumentation

opentelemetry-instrumentation-openai-v2

OpenTelemetry Official OpenAI instrumentation

opentelemetry-langchain-openai-embeddings

OpenTelemetry langchain_openai.embeddings instrumentation

opentelemetry-semantic-conventions-llm

OpenTelemetry Semantic Conventions Extension for Large Language Models

orbitrage

One-line observability + intelligent LLM routing for Python agents.

orchestra-llm-cost

Plug-and-play LLM token/cost tracking SDK with multiple sinks and audit metadata

otel-instrumentation-claude-agent-sdk

OpenTelemetry instrumentation for the Anthropic Claude Agent SDK

overmind-instrumentation-anthropic

OpenTelemetry Anthropic instrumentation

overmind-instrumentation-bedrock

OpenTelemetry Bedrock instrumentation

overmind-instrumentation-cohere

OpenTelemetry Cohere instrumentation

overmind-instrumentation-crewai

OpenTelemetry crewAI instrumentation

overmind-instrumentation-groq

OpenTelemetry Groq instrumentation

overmind-instrumentation-haystack

OpenTelemetry Haystack instrumentation

overmind-instrumentation-langchain

OpenTelemetry Langchain instrumentation

overmind-instrumentation-llamaindex

OpenTelemetry LlamaIndex instrumentation

overmind-instrumentation-mistralai

OpenTelemetry Mistral AI instrumentation

overmind-instrumentation-ollama

OpenTelemetry Ollama instrumentation

overmind-instrumentation-openai

OpenTelemetry OpenAI instrumentation

overmind-instrumentation-openai-agents

OpenTelemetry OpenAI Agents instrumentation

overseex-autogen

OverseeX integration for Microsoft AutoGen - automatic trace capture for multi-agent conversations

overseex-crewai

OverseeX integration for CrewAI - automatic trace capture for multi-agent workflows

overseex-langchain

OverseeX integration for LangChain - automatic trace capture with multi-agent support

pentatonic-agent-events

LLM observability SDK — track token usage, tool calls, and conversations via Pentatonic TES

pentatonic-ai-agent-sdk

AI Agent SDK — observability, memory, and analytics for LLM applications. Provider-agnostic. Tracks token usage, tool calls, conversations, and enables shared team memory.

ph-llmops

Naive LLMOps utilities for quick experimentation

phionyx-langchain-langgraph

Native LangChain + LangGraph adapters for Phionyx runtime evidence — every chain, tool call, and LLM invocation emits a hash-chained envelope entry (signed when a signer is configured; unsigned by default).

phionyx-openai-agents

OpenAI Agents SDK tracing bridge for Phionyx runtime evidence — every Trace and Span emitted by the SDK becomes a hash-chained envelope entry (signed when a signer is configured; unsigned by default).

piqrypt-autoGen-integration

PiQrypt cryptographic audit trail bridge for Microsoft AutoGen

piqrypt-crewai-integration

PiQrypt cryptographic audit trail bridge for CrewAI

pisama-agent-sdk

ATIF trace adapters and real-time failure hooks for agent runtimes (thin shim onto pisama.agents)

pisama-claude-code

Trace capture for Claude Code sessions - sync to Pisama platform for analysis

More in Agent frameworks & infrastructure