Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 54 of 57
Record and query AI agent/tool/service interactions. Shared interaction history for the multi-agent ecosystem.
Generated TypeScript types and service descriptors for the O11y One AGENTIC domain (agentic + common only). A per-domain subset of @o11y-one/api so agentic-only consumers do not carry the other domains. Generated from proto at PROTO_PIN — do not edit by h
LangGraph tracing for Databricks MLflow (TypeScript)
Agent skill for debugging LangChain and LangGraph applications by fetching and analyzing LangSmith traces.
Privacy-first runtime telemetry connector for Agent Coherence.
Renamed to Tracyn -- pip install tracyn instead. See https://tracyn.online
Leave-one-out ablation testing, backward elimination, and ROI evaluation for multi-agent decision systems — find out which agents' findings actually change the outcome.
Agent Conformance Engine (ACE) - deterministic conformance evidence for AI agents. Placeholder release; see the repository for the specification and roadmap.
Framework-agnostic Python library for diagnosing LLM agent execution failures
Evidence-preserving evaluation of complete agent systems
Generate guardrail evaluation plans, JSONL suites, and remediations from agent contracts
Deterministic benchmark harness for command-line coding agents
Conformance corpora and reference verifiers for agent evidence records
See what your AI agents are actually doing. Local-first tracing and debugging for AI agents.
Plug any AI agent into real evaluation scoring, guardrails, and observability tracing in minutes — framework-agnostic, 100% free, no signup, never touches your API key.
LangGraph producer for agent-topology documents
Time-travel debugging and causal RCA for LangGraph agent traces, on a causal graph.
Generate and share self-contained HTML views of agent transcripts
Inspect and improve AI agents with measured fixes and custom scores.
Evidence emitter library for the Agent Conformance Engine (canonical CloudEvents + JSON-LD). Placeholder release.
Plug-and-play turn-metrics logging for LangGraph/Strands/CrewAI agents on Bedrock AgentCore, shaped for CloudWatch Logs Insights + Grafana dashboards.
Application performance monitoring for AI agents: flows, cost, reliability and process quality
Run, watch, and meter Claude Code, Codex CLI, Gemini CLI, OpenCode, and Kiro from one CLI — verified cost tracking, token usage, budget caps, OTel gen_ai spans, Langfuse traces, and a live dashboard + MCP server.
A standalone, framework-agnostic evaluation and scoring engine for LLM/agent outputs — extracted from AgenticLens's evaluation module.
AgentMeasure — check-up report for coding agent runs + conformance pack for telemetry fixture verification. Zero external dependencies.
Replay recorded agent tasks against an OpenAI-compatible endpoint.
Optional Rust streaming client for agentperf-local.
Backend-neutral governance telemetry for AI-agent runtimes
A stress-test benchmark that provokes MAST failure modes in LLM agent frameworks, and shows how much of the failure comes from the framework's own prompt scaffolding.
Anthropic infrastructure FinOps SDK — part of the Hyperscaler Radar suite.
Records what a LangChain agent's tool calls changed on disk, for Apache SkyWalking AI Sessionizer
Avaliação provider-neutral para o Atlas Agent Framework
Use Jev instead of an LLM for RAG decisions: relevance grading, query routing, hallucination checks and LLM-as-a-judge evals. Faster, cheaper, measured first.
Official Python telemetry & remote-config client for CommB (Commercial Bots)
A Spec-Driven Behavioral Contract & Verification Engine for Tool-Using AI Agents
CRANE-LLM: runtime-augmented crash prediction and diagnosis for ML notebooks
Verifiable AER-1 execution receipts for CrewAI agents. Every task event publishes a verifiable receipt at zambo.dev.
Release gates for AI agent output: sources, baselines, statistics, stability, contradictions and adversarial review.
e-volv Observer for Agents SDK for Python: agent runs, model calls and tool calls as spans the Observer rollup reads, with OpenAI, Anthropic and MCP wrappers.
Gemini API infrastructure FinOps SDK — part of the Hyperscaler Radar suite.
Proxy Gemini (Vertex AI) completions wrapped in OpenTelemetry trace spans; returns the answer plus the trace/span id.
2026 OpenAI API Latency & Model Diagnostics CLI (o1/o3/GPT-5/6)
Local Graph-View-Agent for inspecting and replaying LangGraph pipelines
A lightweight, offline Python library for scoring LLM response quality. This project is made under the INFERENCE Lab engineering cohort 01.
Agent tracing SDK aligned with JEP Core 0.7; companion chain semantics remain outside Core
Replay Claude agent sessions with JEP-compatible accountability events, tool lineage, delegation flow, and verification state.
JEP runtime adapter for LangGraph: replayable delegation and verifiable AI accountability semantics.
JEP middleware for OpenAI Agents SDK portable accountability semantics.
Claude Agent SDK recording and replay adapter for Kitaru.
LangChain tool for the Council of AI GSPC measurement board (wraps csoai-gspc).
Thin LangChain integration for Komplex AI — a hallucination detector for LLM output (adapts the halu SDK to LCEL).
A LangChain integration for TypeSafe classifiers
A provider-neutral explainability layer for LangGraph applications.
Standalone LLM cost estimation: model pricing data + token cost math, extracted from LiteLLM with zero litellm dependency.
Universal, provider-agnostic local LLM response analyzer with SQLite storage and a local dashboard.
Provider-agnostic LLM discovery, stability testing and AIPerf benchmarking
LLM cost attribution per feature — track which parts of your product are spending your AI budget
A lightweight, offline Python library for scoring LLM response quality. This project is made under the INFERENCE Lab engineering cohort 01.
Lightweight diagnostic linter for LLM extractions (catches dropped braces, markdown leaks, and hallucinations).
A Python library to visualize LLM perplexity metrics in interactive graphs
Provider-neutral Python instrumentation, portable execution artifacts, and an MCP interface for the Refract execution engine.
Tag LLM spend the way cloud spend is tagged, with calibrated decision models, and export it as FOCUS 1.4
Generic LLM usage and observability framework
OTLP-first Mistral AI observability SDK
Generated Python protobuf messages and Connect clients for the O11y One AGENTIC domain (agentic + common only). A per-domain subset of o11y-one-api so agentic-only consumers do not carry the other domains. Generated from proto at PROTO_PIN — do not edit by hand.
Verifiable execution receipts for OpenAI Agents SDK agents. Every tool call mints a verifiable AER-1 receipt at zambo.dev.
OpenTelemetry Amazon Bedrock instrumentation
OpenTelemetry Official DSPy instrumentation
OpenTelemetry LlamaIndex instrumentation
Record a LangGraph run's own structure — which node ran, in which superstep, and how it ended — into an OrcaReplay trace.
Record the OpenAI Agents SDK's own run structure — agents, handoffs, guardrails — into an OrcaReplay trace.
Pinpoint APM Python agent
Record and replay LLM HTTP traffic for deterministic pytest runs
pyTracer adapter for AutoGen - capture agent runs via an OpenTelemetry exporter.
pyTracer adapter for CrewAI - auto-instrument crews via the event bus.
pyTracer adapter for LangGraph / LangChain - auto-instrument agents via a callback handler.
pyTracer adapter for the OpenAI Agents SDK - auto-instrument agents via a tracing processor.
Score any RAG pipeline: retrieval precision + answer faithfulness
Generic observability framework for RAG applications
RAGAS-style RAG evaluation with JEV as the primary judge
Stage-by-stage diagnostics for RAG pipelines: builds its own eval set, scores retrieval, ranking and generation separately, and tells you which stage is the bottleneck.
Prometheus/Grafana/Loki/AlertManager observability Helm chart for RagLeap Core; the changelog states exactly what each release verified
Lightweight tracing for RAG pipelines
Framework-agnostic RAG evaluation: zero lock-in, zero forced API cost.
Dead-simple regression testing for LLM prompts, built for CI
A reproducibility linter, experiment recorder, lockfile system, and drift detector for LLM research.
Replay AI agent failures locally and turn them into regression tests.
Opt-in vLLM scheduling-context export for TraceLoom AugDB
An LLM-agent evaluation harness that proves an agent is broken when its outputs look fine.
LLM observability and resilience decorator — semantic prompt caching, automatic retries, rate limiting, cost tracking, and a local web dashboard for any Python LLM call.
Durable Vantage lifecycle callback for LiteLLM
Python SDK for Warpswarm: on-demand human judgment for AI evaluation and labeling.
Claude Code usage tracker and daily work log: tokens and hours per project, a timesheet and plain-language summaries, across accounts and computers. Private by default.
Verifiable execution receipts for Haystack agents. Every tool call mints a verifiable AER-1 receipt at zambo.dev.
Verifiable execution receipts for LlamaIndex agents. Every tool call mints a verifiable AER-1 receipt at zambo.dev.
Verifiable execution receipts for OpenAI Agents SDK agents. Every tool call mints a verifiable AER-1 receipt at zambo.dev.
Verifiable execution receipts for smolagents agents. Every tool call mints a verifiable AER-1 receipt at zambo.dev.
Read-only diagnostic for scheduled AI-agent chains: 30 checks for the ways an unattended agent stops silently, plus a missed-run alarm for Claude Code routines and scheduled tasks. One shell file, no dependencies.
Reproducible probes from a long-running AI agent society: memory injection, reasoning budgets, timeouts. Stdlib Python, pre-registered, negative results included.
Agent-12 — the local agent leaderboard. Filesystem-judged agent tasks on consumer hardware, with reference-validated judges.