Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 43 of 57

spline-agent

Spline agent for Python. Lineage tracking utility.

splunk-otel-instrumentation-crewai

OpenTelemetry CrewAI instrumentation

splunk-otel-instrumentation-langchain

OpenTelemetry Official Langchain instrumentation

splunk-otel-instrumentation-llamaindex

Splunk OpenTelemetry LlamaIndex instrumentation

splunk-otel-instrumentation-openai

OpenTelemetry Official OpenAI instrumentation

splunk-otel-instrumentation-openai-agents

Splunk OpenTelemetry OpenAI Agents instrumentation

spryngtime-azure-openai-python

Spryngtime Usage Analytics & Billing API

spryngtime-openai-python

Spryngtime Usage Analytics & Billing API

spyllm

Framework-agnostic agent observability & LLM tracing. Supports OpenAI, Anthropic, CrewAI, AutoGen, LangGraph, and OpenTelemetry.

sr-agent-replay

Dead-simple, local-first, framework-agnostic agent observability. One decorator, one SQLite file, full tracing.

stakeout-agent

Drop-in observability for LangGraph and CrewAI — captures every run, node, tool call, token count, prompt, and response into MongoDB, PostgreSQL, or any OpenTelemetry-compatible collector

strands-agents-evals

Evaluation framework for Strands

streamlit-crewai-process-output

A component for displaying CrewAI agent process outputs in Streamlit applications

stubllm

Deterministic mock server for LLM APIs. Test your AI code without spending tokens.

substr8-langgraph

LangGraph adapter for Substr8 RunProof generation

superopenai

Logging and caching superpowers for the openai sdk

swarm-debug

A colorized, toggleable debug logger with a web GUI

swarm-explainer

Model-agnostic explanations using feature perturbations

swarm-task

Lightweight task orchestration for iterative AI evaluation loops

swarmauri_evaluator_abstractmethods

An evaluator that verifies all methods in abstract base classes are properly marked as abstract

swarmauri_evaluator_anyusage

An evaluator component that detects and penalizes usage of the `Any` type in Python code

swarmauri_evaluator_externalimports

Evaluator that detects and penalizes non-built-in Python imports in source files

swarmauri_evaluator_subprocess

A subprocess-based evaluator for executing and measuring program performance in isolated subprocesses

swarmauri_evaluatorpool_accessibility

A package providing accessibility and readability evaluators for SwarmAuri

swarmauri_measurement_mutualinformation

Swarmauri Mutual Information Measurement Community Package.

swarmauri_measurement_tokencountestimator

This repository includes an example of a First Class Swarmauri Example.

swarmauri_metric_hamming

Swarmauri Hamming distance metric community package.

swarmauri_middleware_time

Middleware for tracking request processing time in Swarmauri applications

swarmauri_tool_psutil

Swarmauri Psutil Tool.

swarmeq

Reserved namespace for swarmeq — affective-state observability for AI agents and multi-agent systems.

swarms-evals

Paper - Pytorch

swarmtrace

pytest for AI agents — trace, debug and catch regressions in LLM swarms

syntrac-opentelemetry-instrumentation-anthropic

OpenTelemetry Anthropic instrumentation

syntrac-opentelemetry-instrumentation-bedrock

OpenTelemetry Bedrock instrumentation

syntrac-opentelemetry-instrumentation-cohere

OpenTelemetry Cohere instrumentation

syntrac-opentelemetry-instrumentation-haystack

OpenTelemetry Haystack instrumentation

syntrac-opentelemetry-instrumentation-langchain

OpenTelemetry Langchain instrumentation

syntrac-opentelemetry-instrumentation-llamaindex

OpenTelemetry LlamaIndex instrumentation

syntrac-opentelemetry-instrumentation-openai

OpenTelemetry OpenAI instrumentation

szn-llm-benchmark-runner

Security placeholder package

tapm-hook-langchain

TAPM v2 hook for langchain (migrated to tapm.hook.langchain namespace)

tapm-hook-llamaindex

TAPM v2 hook for LlamaIndex (migrated to tapm.hook.llamaindex namespace)

tapm-hook-ollama

TAPM v2 hook for ollama (migrated to tapm.hook.ollama namespace)

tapm-hook-openai-v2

TAPM v2 hook for openai_v2 (migrated to tapm.hook.openai_v2 namespace)

tapm-hook-vllm

TAPM v2 hook for vllm (migrated to tapm.hook.vllm namespace)

tapm-instrumentation-langchain

OpenTelemetry Langchain instrumentation

tapm-instrumentation-llamaindex

OpenTelemetry LlamaIndex instrumentation

tapm-instrumentation-ollama

OpenTelemetry Ollama instrumentation

tapm-instrumentation-openai

OpenTelemetry OpenAI instrumentation

tapm-instrumentation-vllm

OpenTelemetry vLLM instrumentation

tessera-agents

Reflect on how you actually use AI coding agents. Reads your local Claude Code, Codex, and Gemini CLI traces, builds rich per-session narratives, and synthesizes cross-session patterns you can act on this week — with an interactive HTML dashboard.

testllm

Deep eval provides evaluation platform to accelerate development of LLMs and Agents

theaios-agent-monitor

Governance-first observability for AI agents

think-dashboard-agent
tianhe-ragas

tianhe ragas modules

timewarp-llm

Event-sourced recording + deterministic replay + time‑travel debugger for LLM agents.

tkm-traceagent

Lightweight observability SDK — traces, spans, decorators, MCP server, and dashboard

tknops-llm

Precise Token Tracking SDK for tknOps

tonic-ragas-logger

Uploads results from ragas to Tonic Validate.

trace-agent-langchain

LangChain callback integration for TraceAgent.

trace-agent-sdk

TraceAgent SDK for instrumenting AI agents.

trace-agent-server

TraceAgent FastAPI backend server.

trace-agent-ui

TraceAgent UI static server.

traceAI-DSPy

OpenTelemetry instrumentation for DSPy

traceAI-anthropic

OpenTelemetry instrumentation for Anthropic

traceAI-autogen

OpenTelemetry instrumentation for AutoGen AgentChat and legacy v0.2

traceAI-bedrock

OpenTelemetry instrumentation for Bedrock

traceAI-cohere

OpenTelemetry instrumentation for Cohere - embeddings, chat, and rerank

traceAI-crewai

OpenTelemetry instrumentation for CrewAI

traceAI-groq

OpenTelemetry instrumentation for Groq

traceAI-haystack

OpenTelemetry instrumentation for Haystack

traceAI-langchain

OpenTelemetry instrumentation for LangChain

traceAI-litellm

OpenTelemetry instrumentation for LiteLLM

traceAI-llamaindex

OpenTelemetry instrumentation for LlamaIndex

traceAI-mistralai

OpenTelemetry instrumentation for Mistral AI

traceAI-ollama

OpenTelemetry instrumentation for Ollama - local LLM runner

traceAI-openai

OpenTelemetry instrumentation for OpenAI

traceAI-openai-agents

OpenTelemetry instrumentation for OpenAI Agents

traceAI-smolagents

OpenTelemetry instrumentation for smolagents

traceAI-vllm

OpenTelemetry instrumentation for vLLM

traceagent

TraceAgent — Local-first AI agent observability SDK. Trace LangChain, LangGraph, and any Python code. Open-source alternative to LangSmith.

traceagently

Real-time observability for AI agents. See every thought, tool call, and error — live.

traceai-claude-agent-sdk

OpenTelemetry instrumentation for Claude Agent SDK

traceai-pydantic-ai

OpenTelemetry instrumentation for Pydantic AI

traceforge-llm

Agent runtime tracing and deterministic replay for LLM applications

tracellm

Local-first observability for LLM applications

tracerazor-anthropic

Drop-in TraceRazor-instrumented Anthropic client. Zero-friction token efficiency auditing.

tracerazor-crewai

TraceRazor callback adapter for CrewAI - automatic token efficiency auditing

tracerazor-langgraph

TraceRazor LangGraph/LangChain callback - token efficiency auditing

tracerazor-openai

Drop-in TraceRazor-instrumented OpenAI client. Zero-friction token efficiency auditing.

tracerazor-openai-agents

TraceRazor hooks adapter for the OpenAI Agents SDK

traceseal-langchain

Signed execution receipts for every LangChain model and tool call.

tracewell-agent

Placeholder package name reserved to protect the official Tracewell project.

tracllm

A context tracing tool for LLM

tracllmkit

A context traceback tool for LLM

trainloop-llm-logging

TrainLoop LLM Logging SDK for data collection

trelis-openai-logger

PostgreSQL logger for OpenAI-style LLM API interactions

triage-agent

Classify why your agent failed. Recover intelligently.

trinity-agent-monitoring

A Python SDK for Trinity Agent Monitoring API

trulens-apps-langchain

Library to systematically track and evaluate LLM based applications.

More in Agent frameworks & infrastructure