Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 50 of 57

agento11y-openai

OpenAI helper wrappers for the Grafana Agent Observability Python SDK

agento11y-openai-agents

OpenAI Agents callback handlers for the Grafana Agent Observability Python SDK

agento11y-pydantic-ai

Pydantic AI capability for the Grafana Agent Observability Python SDK

agento11y-strands

Strands Agents hooks for the Grafana Agent Observability Python SDK

agentops-otel

Shared OpenTelemetry bootstrap for AgentOps (control plane + SDK workers).

agentpoke

Drop-in pytest plugin for regression-testing AI agents — snapshot baselines, semantic comparison, mock LLMs

agentqa-sdk

Regression testing and monitoring for AI agents — record every run, test every change.

agentrange

AgentRange — CLI + SDK for reproducible security-agent evals.

agentrons

Agentron (Ag): a unit of agentic work volume — Ag = hours × Mtokens.

agentrubric

AgentRubric: trajectory-level evaluation for LLM agents

agenttic-langgraph

Trace a LangGraph agent onto the Agenttic OTel-GenAI bus (public callbacks only).

agenttic-openai-agents

Trace an OpenAI Agents SDK agent onto the Agenttic OTel-GenAI bus (public RunHooks only).

agenttrace-langchain

Deprecated shim — use lanetrace[langchain] (import lanetrace.langchain)

agentverity

Qualify repeated categorical AI-agent evidence before it becomes a regression reference

agentwatch-dev

Predict when production AI agents degrade — before they visibly fail. Python SDK.

agentwatch-observe

Lightweight observability SDK for AI agents — wrap any agent framework, get anomaly detection, guardrails, and cost tracking.

agentworkflow-viz

Agent workflow graph visualization plugin for LangGraph and other frameworks

aurora-agent

Local-first runtime evidence SDK with a self-serve AURORA quickstart

azure-mgmt-monitoragents

This package will be released in the near future. Stay tuned!

btech-agent-lab

Cloud-native LLM Agent Development and Testing Toolkit

cardinal-agent-core

Shared Cardinal agent runtime: telemetry contracts.

checkyouragent

Local analytics for Claude Code project exports

chronicle-agent-sdk

Python SDK for capturing and shipping AI agent traces to a Chronicle server

claude-usage-exporter

Export Claude Code token usage (per agent/model) to Prometheus, OTLP, or stdout -- parsed from transcripts, works on Max/Pro subscriptions.

claude-usage-tui

Local Claude Code token activity graphs with explicit source accuracy

clickllm-cli

Prove which open model can replace your closed one — on your traffic, your hardware, your budget.

coding-agent-vcr

VCR / time-travel replay for Claude Code and Codex — deterministic offline agent testing

confident-agent

Relay agent for bridging Confident AI evaluation server to internal AI connections endpoints

ctxvsrag

Benchmark full-context-in-prompt vs. RAG for a document, against Ollama, vLLM, LiteLLM, Open WebUI, or any OpenAI-compatible backend

dashboard-for-claude-code

Claude Code Personal Analytics — a private, local, read-only analytics dashboard for your Claude Code sessions: costs, tokens, tools, skills, coding patterns, and wasted effort.

deepagents-trace

Deprecated: renamed to lanetrace. This package installs lanetrace and keeps `import agenttrace` and the `agenttrace` command working.

dhrions-llm-usage

Tarification, comptage de tokens et affichage LLM partagés du parc

dnor-agent-meter

Lightweight SDK to track AI agent tool calls, costs, and latency.

dnorio-agent-meter

Lightweight SDK to track AI agent tool calls, costs, and latency.

edd-agent-tools

Evaluation-Driven Development (EDD) tools and helper libraries for AI Agent development

eval-agents

AgentLab — a modular framework for building, evaluating, and comparing LLM-powered agents.

evallm-cli

LLM evaluation framework for developers

evalragkit

Composable RAG evaluation library for experiment-based testing

fragility-engine

AgenticOps stress-search harness: deterministic GA/MC over modular simulations, BYOW adapters, replay JSON, and FEL evidence semantics.

gensimllms

Python library to support LLM testing.

gosce-langsmith-traced-llm-orchestration-agent

Run a prompt through a LangChain (system + human) chain over Gemini on Vertex AI; optional LangSmith tracing.

gosce-llm-observability-orchestration-agent-langchain-community

Run a prompt through a LangChain (system + human) chain over Gemini on Vertex AI; optional LangSmith tracing.

gosce-llm-orchestration-agent-langsmith

Run a prompt through a LangChain (system + human) chain over Gemini on Vertex AI; optional LangSmith tracing.

gosce-llm-orchestration-observability-agent

Run a prompt through a LangChain (system + human) chain over Gemini on Vertex AI; optional LangSmith tracing.

gosce-traced-llm-proxy-anthropic

Proxy Gemini (Vertex AI) completions wrapped in OpenTelemetry trace spans; returns the answer plus the trace/span id.

graphsight-langgraph

X-ray for LangGraph retrieval: one callback handler captures every retriever call, score, and relational path — rendered as an interactive graph.

hermes-llm-api-call-logger

Log every LLM API call with full request/response details for Hermes Agent

hyperprobe-agent

Production-grade, non-breaking live debugger and telemetry agent for Python.

komodor-agentops

Python SDK for AgentOps observability.

litellm-openttt

TTTPS Proof-of-Time success callback for LiteLLM: cryptographic audit-trail timestamps for LLM completions via the public KPP Provenance API

llama-parameter-scout

Benchmark llama.cpp models across candidate configurations

llamaindex-openttt

TTTPS Proof-of-Time callback handler for LlamaIndex: cryptographic audit-trail timestamps for LLM events via the public KPP Provenance API

llamathon

Benchmark the LLM inference capacity of a server (llama-benchy orchestrator with auto-detection, real-workload sizing, charts and reports).

llm-agents-from-scratch-inspector

A dev tool for manually driving llm-agents-from-scratch's SupervisedTaskHandler one call at a time, over HTTP, via a React frontend.

llm-audit-trail

Tamper-evident audit trails for LLM lifecycles: training, deployment, and monitoring.

llm-code-benchmark

OpenRouter LLM benchmark for repair-agent and validator-agent model selection.

llm-comply

Multi-format LLM API compliance testing tool — validates endpoints against OpenAI Chat, Open Responses, Anthropic, and Google GenAI specs.

llm-eval-exporter

A lightweight Prometheus exporter for LLM eval metrics: faithfulness, semantic drift, cost, and CI/CD regression gating.

llm-inspector

The htop for LLM inference. Measured. Not guessed.

llm-judge-panel

A multi-provider LLM-as-a-Judge consensus panel with chained criteria for concept matching validation.

llm-log-system

Self-hosted LLM observability SDK for Python, LangChain, and LangGraph

llm-observer-proxy

Run-scoped LLM observability control plane backed by LiteLLM

llm-preflight

The pre-merge check for LLM model, prompt, schema, and provider changes

llm-prompt-radar

A GitHub Action and CLI that detects risky changes to LLM prompts and AI configuration before they ship to production

llm-run-recorder

Flight recorder for LLM agents: trace every call, replay runs deterministically, diff runs to find where behavior diverged.

llm-sketchkit

Count distinct LLM users and find top token consumers across services, days and tools. Fast, small, portable summaries in Go and Python.

llm-speed-bench

Local smoke tests for live LLM models, latency, and cost

llm-token-guard

Track LLM token usage, enforce policies & budget limits, and trigger alerts — across OpenAI, Groq, OpenRouter, AWS Bedrock, and custom providers.

llm-token-ledger

Per-query token ledger for LLM pipelines: record what the API reports, answer with one GROUP BY. Zero dependencies, dashboard included.

llm-token-lens

Local LLM token efficiency middleware — zero cloud, zero tracking, coaches developers on token waste in the terminal

llm-tracker

Track LLM API costs, tokens, and latency to MySQL

llm-usage-meter

Lightweight terminal tool to check remaining credits / quota across LLM providers. Zero dependencies.

llm-verifier

A general-purpose LLM verification framework: fine-grained reward + Pivot Preference Tournament best-of-N selection.

llmintent

LLMIntent suite: Anatomy of any weighted LLM, fly-connectome prior, trajectory, MisAlign Flag, isolates, IV, latent inspect

llmlimits

Python library to support LLM testing.

llmrunctl

LLM Runner Control is a CLI tool to interact with local and remote LLM, define multiple scenario variations (model × prompt × options), run it over a dataset and capture structured results for comparison or to build knowledge through meta analysis.

llmtrafficlens

Profile LLM gateway logs for prefix-cache reuse potential, and export anonymized replayable traces

llmwho

Passive LLM endpoint identity and stability instrumentation

mosaic-llm-bench

MOSAIC: evaluating the Moral, Social and Individual dimensions of LLMs

nishanttyagi-agenteval

CI-integrated evaluation harness for LLM agents

observation-agent

Agent 观测 SDK:模块四段(输入/结构/流程/输出)申报 + LangChain 自动采集 + HTTP 上报

openagent-eval

Open-source CLI framework for evaluating RAG systems and AI Agents

openai-token-monitor

A self-metering guardrail that counts and caps your OpenAI token usage against a daily (UTC) cap.

openaimeter

A vendor-neutral FinOps and measurement toolkit for AI usage, cost, efficiency, and business outcomes.

openeval-llm

Lightweight LLM-as-judge evaluation CLI — score model outputs on 5 dimensions and compare runs (OpenAI / OpenRouter / Ollama)

opentelemetry-instrumentation-genai-anthropic

OpenTelemetry Anthropic instrumentation

opentelemetry-instrumentation-genai-langchain

OpenTelemetry Official Langchain instrumentation

opentelemetry-instrumentation-genai-openai

OpenTelemetry Official OpenAI instrumentation

opentelemetry-instrumentation-genai-openai-agents

OpenTelemetry Official OpenAI Agents instrumentation

orqis-agent-sdk

Orqis SDK - one line to stream AI agent traces to Orqis self-healing ops

pluto-llm-diff

git diff, but for LLM behavior — behavioral regression testing for language models.

pydantic-ai-trace

Fast local web viewer for pydantic-ai traces (list[ModelMessage] JSON dumps)

pyllmits

Large Language Model Instinct Testing under Survival - benchmark LLMs in a hand-built Crafter survival world.

quorumgate-llm

Never ship unverified LLM output: multi-model councils, severity-graded audit gates, bounded retries, deterministic fallbacks. Zero dependencies.

rag-audit-toolkit

Audit toolkit for retrieval, evaluation, benchmark, and security checks in RAG systems

rag-debug-cli

A debugger for RAG applications: diagnoses retrieval, context, and grounding failures.

rag-failcase-kit

Lightweight JSONL tracing for debugging failed RAG answers.

rag-scorer

A from-scratch CLI evaluation framework for RAG pipelines.

ragaeval

A pip-installable Python library providing drop-in RAG evaluation capabilities using RAGAS

raggate

A thin, CI-gated evaluation gate for RAG & LLM systems: golden set, LLM-judge (or heuristic) scorers, and band-based quality gates.

More in Agent frameworks & infrastructure