Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 40 of 57
OpenAI cost tracker that logs OpenAI requests using opentelemetry
🧪🤖 Pytest plugin for automatically mocking OpenAI requests
Quick CLI to test OpenAI-compatible API endpoints.
OpenAI Token Counter
Simple Library for OpenAI Usage
A utility for logging OpenAI API usage and costs.
Console OpenAI API cost and token usage analysis by key and model
Calculate costs for OpenAI vision model image inputs
A wrapper for OpenAI's python API which wraps around the openAI functions and stores the request, response and metadata to MongoDB. The stored data can be used to fine tune GPT-3 models or HuggingFace models.
Universal LLM introspection. Understand any model. Any architecture.
latest pricing information of openai models, crowd maintained
Add your description here
OpenBox governance SDK for DeepAgents (langchain-ai/deepagents)
OpenBox governance and observability SDK for LangChain
OpenBox governance and observability SDK for LangGraph
OpenCensus OC-Agent Trace Exporter
Node agent for OpenClaw Central distributed monitoring platform
OpenInference span processor for Microsoft Agent Framework - transforms native OpenTelemetry spans to OpenInference format
OpenInference auto-instrumentation for the AgentMinds Python SDK. Emits OTel spans tagged with openinference.span.kind + agentminds.* attributes when agentminds.capture_* and agentminds.init are called.
OpenInference Open Agent Spec Instrumentation
OpenInference Anthropic Instrumentation
Deprecated compatibility package for OpenInference AG2 instrumentation
OpenInference Autogen-Agentchat Instrumentation
OpenInference Bedrock Instrumentation
OpenInference Claude Agent SDK Instrumentation
OpenInference Crewai Instrumentation
OpenInference DSPy Instrumentation
OpenInference Groq Instrumentation
OpenInference Haystack Instrumentation
OpenInference LangChain Instrumentation
OpenInference liteLLM Instrumentation
OpenInference LlamaIndex Instrumentation
OpenInference Mistral AI Instrumentation
OpenInference instrumentation for MonkAI Agent
OpenInference OpenAI Instrumentation
OpenInference OpenAI Agents Instrumentation
OpenInference OpenLLMetry Instrumentation
OpenInference PydanticAI Instrumentation
OpenInference smolagents Instrumentation
OpenInference span processor for Strands Agents - transforms native OpenTelemetry spans to OpenInference format
End-to-end observability with built-in security guardrails.
OpenTelemetry instrumentation for the Anthropic Claude Agent SDK
OpenTelemetry Anthropic instrumentation
OpenTelemetry Bedrock instrumentation
OpenTelemetry Cohere instrumentation
OpenTelemetry Haystack instrumentation
OpenTelemetry Langchain instrumentation
OpenTelemetry LlamaIndex instrumentation
OpenTelemetry Mistral AI instrumentation
OpenTelemetry Ollama instrumentation
OpenTelemetry OpenAI instrumentation
Instruments Anthropic client for OpenTelemetry
OpenTelemetry openai instrumentation
OpenTelemetry Anthropic instrumentation
OpenTelemetry Bedrock instrumentation
OpenTelemetry Bedrock instrumentation
OpenTelemetry Cohere instrumentation
OpenTelemetry crewAI instrumentation
OpenTelemetry Groq instrumentation
OpenTelemetry Haystack instrumentation
OpenTelemetry Langchain instrumentation
OpenTelemetry langchain_openai.embeddings instrumentation
OpenTelemetry Langchain instrumentation
OpenTelemetry LlamaIndex instrumentation
OpenTelemetry Mistral AI instrumentation
OpenTelemetry Ollama instrumentation
OpenTelemetry OpenAI instrumentation
OpenTelemetry OpenAI Agents instrumentation
OpenTelemetry OpenAI Agents instrumentation (barebones)
OpenTelemetry OpenAI instrumentation
OpenTelemetry Official OpenAI instrumentation
OpenTelemetry langchain_openai.embeddings instrumentation
OpenTelemetry Semantic Conventions Extension for Large Language Models
One-line observability + intelligent LLM routing for Python agents.
Plug-and-play LLM token/cost tracking SDK with multiple sinks and audit metadata
OpenTelemetry instrumentation for the Anthropic Claude Agent SDK
OpenTelemetry Anthropic instrumentation
OpenTelemetry Bedrock instrumentation
OpenTelemetry Cohere instrumentation
OpenTelemetry crewAI instrumentation
OpenTelemetry Groq instrumentation
OpenTelemetry Haystack instrumentation
OpenTelemetry Langchain instrumentation
OpenTelemetry LlamaIndex instrumentation
OpenTelemetry Mistral AI instrumentation
OpenTelemetry Ollama instrumentation
OpenTelemetry OpenAI instrumentation
OpenTelemetry OpenAI Agents instrumentation
OverseeX integration for Microsoft AutoGen - automatic trace capture for multi-agent conversations
OverseeX integration for CrewAI - automatic trace capture for multi-agent workflows
OverseeX integration for LangChain - automatic trace capture with multi-agent support
LLM observability SDK — track token usage, tool calls, and conversations via Pentatonic TES
AI Agent SDK — observability, memory, and analytics for LLM applications. Provider-agnostic. Tracks token usage, tool calls, conversations, and enables shared team memory.
Naive LLMOps utilities for quick experimentation
Native LangChain + LangGraph adapters for Phionyx runtime evidence — every chain, tool call, and LLM invocation emits a hash-chained envelope entry (signed when a signer is configured; unsigned by default).
OpenAI Agents SDK tracing bridge for Phionyx runtime evidence — every Trace and Span emitted by the SDK becomes a hash-chained envelope entry (signed when a signer is configured; unsigned by default).
PiQrypt cryptographic audit trail bridge for Microsoft AutoGen
PiQrypt cryptographic audit trail bridge for CrewAI
ATIF trace adapters and real-time failure hooks for agent runtimes (thin shim onto pisama.agents)
Trace capture for Claude Code sessions - sync to Pisama platform for analysis