Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 43 of 57
Spline agent for Python. Lineage tracking utility.
OpenTelemetry CrewAI instrumentation
OpenTelemetry Official Langchain instrumentation
Splunk OpenTelemetry LlamaIndex instrumentation
OpenTelemetry Official OpenAI instrumentation
Splunk OpenTelemetry OpenAI Agents instrumentation
Spryngtime Usage Analytics & Billing API
Spryngtime Usage Analytics & Billing API
Framework-agnostic agent observability & LLM tracing. Supports OpenAI, Anthropic, CrewAI, AutoGen, LangGraph, and OpenTelemetry.
Dead-simple, local-first, framework-agnostic agent observability. One decorator, one SQLite file, full tracing.
Drop-in observability for LangGraph and CrewAI — captures every run, node, tool call, token count, prompt, and response into MongoDB, PostgreSQL, or any OpenTelemetry-compatible collector
Evaluation framework for Strands
A component for displaying CrewAI agent process outputs in Streamlit applications
Deterministic mock server for LLM APIs. Test your AI code without spending tokens.
LangGraph adapter for Substr8 RunProof generation
Logging and caching superpowers for the openai sdk
A colorized, toggleable debug logger with a web GUI
Model-agnostic explanations using feature perturbations
Lightweight task orchestration for iterative AI evaluation loops
An evaluator that verifies all methods in abstract base classes are properly marked as abstract
An evaluator component that detects and penalizes usage of the `Any` type in Python code
Evaluator that detects and penalizes non-built-in Python imports in source files
A subprocess-based evaluator for executing and measuring program performance in isolated subprocesses
A package providing accessibility and readability evaluators for SwarmAuri
Swarmauri Mutual Information Measurement Community Package.
This repository includes an example of a First Class Swarmauri Example.
Swarmauri Hamming distance metric community package.
Middleware for tracking request processing time in Swarmauri applications
Swarmauri Psutil Tool.
Reserved namespace for swarmeq — affective-state observability for AI agents and multi-agent systems.
Paper - Pytorch
pytest for AI agents — trace, debug and catch regressions in LLM swarms
OpenTelemetry Anthropic instrumentation
OpenTelemetry Bedrock instrumentation
OpenTelemetry Cohere instrumentation
OpenTelemetry Haystack instrumentation
OpenTelemetry Langchain instrumentation
OpenTelemetry LlamaIndex instrumentation
OpenTelemetry OpenAI instrumentation
Security placeholder package
TAPM v2 hook for langchain (migrated to tapm.hook.langchain namespace)
TAPM v2 hook for LlamaIndex (migrated to tapm.hook.llamaindex namespace)
TAPM v2 hook for ollama (migrated to tapm.hook.ollama namespace)
TAPM v2 hook for openai_v2 (migrated to tapm.hook.openai_v2 namespace)
TAPM v2 hook for vllm (migrated to tapm.hook.vllm namespace)
OpenTelemetry Langchain instrumentation
OpenTelemetry LlamaIndex instrumentation
OpenTelemetry Ollama instrumentation
OpenTelemetry OpenAI instrumentation
OpenTelemetry vLLM instrumentation
Reflect on how you actually use AI coding agents. Reads your local Claude Code, Codex, and Gemini CLI traces, builds rich per-session narratives, and synthesizes cross-session patterns you can act on this week — with an interactive HTML dashboard.
Deep eval provides evaluation platform to accelerate development of LLMs and Agents
Governance-first observability for AI agents
tianhe ragas modules
Event-sourced recording + deterministic replay + time‑travel debugger for LLM agents.
Lightweight observability SDK — traces, spans, decorators, MCP server, and dashboard
Precise Token Tracking SDK for tknOps
Uploads results from ragas to Tonic Validate.
LangChain callback integration for TraceAgent.
TraceAgent SDK for instrumenting AI agents.
TraceAgent FastAPI backend server.
TraceAgent UI static server.
OpenTelemetry instrumentation for DSPy
OpenTelemetry instrumentation for Anthropic
OpenTelemetry instrumentation for AutoGen AgentChat and legacy v0.2
OpenTelemetry instrumentation for Bedrock
OpenTelemetry instrumentation for Cohere - embeddings, chat, and rerank
OpenTelemetry instrumentation for CrewAI
OpenTelemetry instrumentation for Groq
OpenTelemetry instrumentation for Haystack
OpenTelemetry instrumentation for LangChain
OpenTelemetry instrumentation for LiteLLM
OpenTelemetry instrumentation for LlamaIndex
OpenTelemetry instrumentation for Mistral AI
OpenTelemetry instrumentation for Ollama - local LLM runner
OpenTelemetry instrumentation for OpenAI
OpenTelemetry instrumentation for OpenAI Agents
OpenTelemetry instrumentation for smolagents
OpenTelemetry instrumentation for vLLM
TraceAgent — Local-first AI agent observability SDK. Trace LangChain, LangGraph, and any Python code. Open-source alternative to LangSmith.
Real-time observability for AI agents. See every thought, tool call, and error — live.
OpenTelemetry instrumentation for Claude Agent SDK
OpenTelemetry instrumentation for Pydantic AI
Agent runtime tracing and deterministic replay for LLM applications
Local-first observability for LLM applications
Drop-in TraceRazor-instrumented Anthropic client. Zero-friction token efficiency auditing.
TraceRazor callback adapter for CrewAI - automatic token efficiency auditing
TraceRazor LangGraph/LangChain callback - token efficiency auditing
Drop-in TraceRazor-instrumented OpenAI client. Zero-friction token efficiency auditing.
TraceRazor hooks adapter for the OpenAI Agents SDK
Signed execution receipts for every LangChain model and tool call.
Placeholder package name reserved to protect the official Tracewell project.
A context tracing tool for LLM
A context traceback tool for LLM
TrainLoop LLM Logging SDK for data collection
PostgreSQL logger for OpenAI-style LLM API interactions
Classify why your agent failed. Recover intelligently.
A Python SDK for Trinity Agent Monitoring API
Library to systematically track and evaluate LLM based applications.