Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 54 of 57

lowdown-proxy

Record and query AI agent/tool/service interactions. Shared interaction history for the multi-agent ecosystem.

@o11y-one/api-agentic

Generated TypeScript types and service descriptors for the O11y One AGENTIC domain (agentic + common only). A per-domain subset of @o11y-one/api so agentic-only consumers do not carry the other domains. Generated from proto at PROTO_PIN — do not edit by h

@insightfactory.ai/insightfactory-databricks-langgraph-tracer

LangGraph tracing for Databricks MLflow (TypeScript)

langsmith-fetch-skill

Agent skill for debugging LangChain and LangGraph applications by fetching and analyzing LangSmith traces.

@coherence_skill/runtime-connector

Privacy-first runtime telemetry connector for Agent Coherence.

AudAgent

Renamed to Tracyn -- pip install tracyn instead. See https://tracyn.online

agent-ablation

Leave-one-out ablation testing, backward elimination, and ROI evaluation for multi-agent decision systems — find out which agents' findings actually change the outcome.

agent-conformance

Agent Conformance Engine (ACE) - deterministic conformance evidence for AI agents. Placeholder release; see the repository for the specification and roadmap.

agent-diagnostician

Framework-agnostic Python library for diagnosing LLM agent execution failures

agent-eval-flow

Evidence-preserving evaluation of complete agent systems

agent-eval-planner

Generate guardrail evaluation plans, JSONL suites, and remediations from agent contracts

agent-evaluation-lab

Deterministic benchmark harness for command-line coding agents

agent-evidence-vectors

Conformance corpora and reference verifiers for agent evidence records

agent-lense

See what your AI agents are actually doing. Local-first tracing and debugging for AI agents.

agent-referee

Plug any AI agent into real evaluation scoring, guardrails, and observability tracing in minutes — framework-agnostic, 100% free, no signup, never touches your API key.

agent-topology-langgraph

LangGraph producer for agent-topology documents

agent-tracegraph

Time-travel debugging and causal RCA for LangGraph agent traces, on a causal graph.

agent-transcript-viewer

Generate and share self-contained HTML views of agent transcripts

agentagon

Inspect and improve AI agents with measured fixes and custom scores.

agentce-emit

Evidence emitter library for the Agent Conformance Engine (canonical CloudEvents + JSON-LD). Placeholder release.

agentcore-dashboard-metrics

Plug-and-play turn-metrics logging for LangGraph/Strands/CrewAI agents on Bedrock AgentCore, shaped for CloudWatch Logs Insights + Grafana dashboards.

agentdynamics

Application performance monitoring for AI agents: flows, cost, reliability and process quality

agentic-coding-harness

Run, watch, and meter Claude Code, Codex CLI, Gemini CLI, OpenCode, and Kiro from one CLI — verified cost tracking, token usage, budget caps, OTel gen_ai spans, Langfuse traces, and a live dashboard + MCP server.

agentic-evals

A standalone, framework-agnostic evaluation and scoring engine for LLM/agent outputs — extracted from AgenticLens's evaluation module.

agentmeasure

AgentMeasure — check-up report for coding agent runs + conformance pack for telemetry fixture verification. Zero external dependencies.

agentperf-local

Replay recorded agent tasks against an OpenAI-compatible endpoint.

agentperf-local-rustcore

Optional Rust streaming client for agentperf-local.

agentrust-telemetry

Backend-neutral governance telemetry for AI-agent runtimes

agentstress

A stress-test benchmark that provokes MAST failure modes in LLM agent frameworks, and shows how much of the failure comes from the framework's own prompt scaffolding.

anthropic-radar

Anthropic infrastructure FinOps SDK — part of the Hyperscaler Radar suite.

apache-skywalking-asz-langchain

Records what a LangChain agent's tool calls changed on disk, for Apache SkyWalking AI Sessionizer

atlas-agent-evaluation

Avaliação provider-neutral para o Atlas Agent Framework

better-cheaper-llm

Use Jev instead of an LLM for RAG decisions: relevance grading, query routing, hallucination checks and LLM-as-a-judge evals. Faster, cheaper, measured first.

commb-agent

Official Python telemetry & remote-config client for CommB (Commercial Bots)

contract-agent

A Spec-Driven Behavioral Contract & Verification Engine for Tool-Using AI Agents

crane-llm

CRANE-LLM: runtime-augmented crash prediction and diagnosis for ML notebooks

crewai-zambo

Verifiable AER-1 execution receipts for CrewAI agents. Every task event publishes a verifiable receipt at zambo.dev.

dankaro-agent-quality

Release gates for AI agent output: sources, baselines, statistics, stability, contradictions and adversarial review.

evolv-agents

e-volv Observer for Agents SDK for Python: agent runs, model calls and tool calls as spans the Observer rollup reads, with OpenAI, Anthropic and MCP wrappers.

gemini-radar

Gemini API infrastructure FinOps SDK — part of the Hyperscaler Radar suite.

gosce-traced-claude-format-guard

Proxy Gemini (Vertex AI) completions wrapped in OpenTelemetry trace spans; returns the answer plus the trace/span id.

gpt-ping-cn

2026 OpenAI API Latency & Model Diagnostics CLI (o1/o3/GPT-5/6)

graphviagent

Local Graph-View-Agent for inspecting and replaying LangGraph pipelines

inferencelab-llm-eval-kit

A lightweight, offline Python library for scoring LLM response quality. This project is made under the INFERENCE Lab engineering cohort 01.

jep-agent-sdk

Agent tracing SDK aligned with JEP Core 0.7; companion chain semantics remain outside Core

jep-claude-replay

Replay Claude agent sessions with JEP-compatible accountability events, tool lineage, delegation flow, and verification state.

jep-langgraph-adapter

JEP runtime adapter for LangGraph: replayable delegation and verifiable AI accountability semantics.

jep-openai-agents-middleware

JEP middleware for OpenAI Agents SDK portable accountability semantics.

kitaru-claude-agent-sdk

Claude Agent SDK recording and replay adapter for Kitaru.

langchain-csoai

LangChain tool for the Council of AI GSPC measurement board (wraps csoai-gspc).

langchain-halu

Thin LangChain integration for Komplex AI — a hallucination detector for LLM output (adapts the halu SDK to LCEL).

langchain-typesafe

A LangChain integration for TypeSafe classifiers

langgraph-xai

A provider-neutral explainability layer for LangGraph applications.

litellm-cost

Standalone LLM cost estimation: model pricing data + token cost math, extracted from LiteLLM with zero litellm dependency.

llm-analyzer

Universal, provider-agnostic local LLM response analyzer with SQLite storage and a local dashboard.

llm-benchmark-manager

Provider-agnostic LLM discovery, stability testing and AIPerf benchmarking

llm-cost-track

LLM cost attribution per feature — track which parts of your product are spending your AI budget

llm-eval-kit-v1

A lightweight, offline Python library for scoring LLM response quality. This project is made under the INFERENCE Lab engineering cohort 01.

llm-extraction-auditor

Lightweight diagnostic linter for LLM extractions (catches dropped braces, markdown leaks, and hallucinations).

llm-pig

A Python library to visualize LLM perplexity metrics in interactive graphs

llm-refract

Provider-neutral Python instrumentation, portable execution artifacts, and an MCP interface for the Refract execution engine.

llm-tokenomics

Tag LLM spend the way cloud spend is tagged, with calibrated decision models, and export it as FOCUS 1.4

llm-usage-framework

Generic LLM usage and observability framework

mistralai-otel-observability

OTLP-first Mistral AI observability SDK

o11y-one-api-agentic

Generated Python protobuf messages and Connect clients for the O11y One AGENTIC domain (agentic + common only). A per-domain subset of o11y-one-api so agentic-only consumers do not carry the other domains. Generated from proto at PROTO_PIN — do not edit by hand.

openai-agents-zambo

Verifiable execution receipts for OpenAI Agents SDK agents. Every tool call mints a verifiable AER-1 receipt at zambo.dev.

opentelemetry-instrumentation-genai-bedrock

OpenTelemetry Amazon Bedrock instrumentation

opentelemetry-instrumentation-genai-dspy

OpenTelemetry Official DSPy instrumentation

opentelemetry-instrumentation-genai-llama-index

OpenTelemetry LlamaIndex instrumentation

orcareplay-langgraph

Record a LangGraph run's own structure — which node ran, in which superstep, and how it ended — into an OrcaReplay trace.

orcareplay-openai-agents

Record the OpenAI Agents SDK's own run structure — agents, handoffs, guardrails — into an OrcaReplay trace.

pinpoint-python-agent

Pinpoint APM Python agent

pytest-llm-vcr

Record and replay LLM HTTP traffic for deterministic pytest runs

pytracer-autogen

pyTracer adapter for AutoGen - capture agent runs via an OpenTelemetry exporter.

pytracer-crewai

pyTracer adapter for CrewAI - auto-instrument crews via the event bus.

pytracer-langgraph

pyTracer adapter for LangGraph / LangChain - auto-instrument agents via a callback handler.

pytracer-openai-agents

pyTracer adapter for the OpenAI Agents SDK - auto-instrument agents via a tracing processor.

rag-eval-py

Score any RAG pipeline: retrieval precision + answer faithfulness

rag-observability

Generic observability framework for RAG applications

ragas-jev

RAGAS-style RAG evaluation with JEV as the primary judge

ragbisect

Stage-by-stage diagnostics for RAG pipelines: builds its own eval set, scores retrieval, ranking and generation separately, and tells you which stage is the bottleneck.

ragleap-observability

Prometheus/Grafana/Loki/AlertManager observability Helm chart for RagLeap Core; the changelog states exactly what each release verified

raglens-sdk

Lightweight tracing for RAG pipelines

ragmark

Framework-agnostic RAG evaluation: zero lock-in, zero forced API cost.

regress-llm

Dead-simple regression testing for LLM prompts, built for CI

reprollm

A reproducibility linter, experiment recorder, lockfile system, and drift detector for LLM research.

tracefork-agent

Replay AI agent failures locally and turn them into regression tests.

traceloom-vllm-context

Opt-in vLLM scheduling-context export for TraceLoom AugDB

trust-no-agent

An LLM-agent evaluation harness that proves an agent is broken when its outputs look fine.

undertow-llm

LLM observability and resilience decorator — semantic prompt caching, automatic retries, rate limiting, cost tracking, and a local web dashboard for any Python LLM call.

vantage-litellm-callback

Durable Vantage lifecycle callback for LiteLLM

warpswarm

Python SDK for Warpswarm: on-demand human judgment for AI evaluation and labeling.

worklog-for-claude-code

Claude Code usage tracker and daily work log: tokens and hours per project, a timesheet and plain-language summaries, across accounts and computers. Private by default.

zambo-haystack

Verifiable execution receipts for Haystack agents. Every tool call mints a verifiable AER-1 receipt at zambo.dev.

zambo-llamaindex

Verifiable execution receipts for LlamaIndex agents. Every tool call mints a verifiable AER-1 receipt at zambo.dev.

zambo-openai-agents

Verifiable execution receipts for OpenAI Agents SDK agents. Every tool call mints a verifiable AER-1 receipt at zambo.dev.

zambo-smolagents

Verifiable execution receipts for smolagents agents. Every tool call mints a verifiable AER-1 receipt at zambo.dev.

agent-chain-doctor

Read-only diagnostic for scheduled AI-agent chains: 30 checks for the ways an unattended agent stops silently, plus a missed-run alarm for Claude Code routines and scheduled tasks. One shell file, no dependencies.

colony-lab-findings

Reproducible probes from a long-running AI agent society: memory injection, reasoning budgets, timeouts. Stdlib Python, pre-registered, negative results included.

agent12

Agent-12 — the local agent leaderboard. Filesystem-judged agent tasks on consumer hardware, with reference-validated judges.

More in Agent frameworks & infrastructure