Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 50 of 57
OpenAI helper wrappers for the Grafana Agent Observability Python SDK
OpenAI Agents callback handlers for the Grafana Agent Observability Python SDK
Pydantic AI capability for the Grafana Agent Observability Python SDK
Strands Agents hooks for the Grafana Agent Observability Python SDK
Shared OpenTelemetry bootstrap for AgentOps (control plane + SDK workers).
Drop-in pytest plugin for regression-testing AI agents — snapshot baselines, semantic comparison, mock LLMs
Regression testing and monitoring for AI agents — record every run, test every change.
AgentRange — CLI + SDK for reproducible security-agent evals.
Agentron (Ag): a unit of agentic work volume — Ag = hours × Mtokens.
AgentRubric: trajectory-level evaluation for LLM agents
Trace a LangGraph agent onto the Agenttic OTel-GenAI bus (public callbacks only).
Trace an OpenAI Agents SDK agent onto the Agenttic OTel-GenAI bus (public RunHooks only).
Deprecated shim — use lanetrace[langchain] (import lanetrace.langchain)
Qualify repeated categorical AI-agent evidence before it becomes a regression reference
Predict when production AI agents degrade — before they visibly fail. Python SDK.
Lightweight observability SDK for AI agents — wrap any agent framework, get anomaly detection, guardrails, and cost tracking.
Agent workflow graph visualization plugin for LangGraph and other frameworks
Local-first runtime evidence SDK with a self-serve AURORA quickstart
This package will be released in the near future. Stay tuned!
Cloud-native LLM Agent Development and Testing Toolkit
Shared Cardinal agent runtime: telemetry contracts.
Local analytics for Claude Code project exports
Python SDK for capturing and shipping AI agent traces to a Chronicle server
Export Claude Code token usage (per agent/model) to Prometheus, OTLP, or stdout -- parsed from transcripts, works on Max/Pro subscriptions.
Local Claude Code token activity graphs with explicit source accuracy
Prove which open model can replace your closed one — on your traffic, your hardware, your budget.
VCR / time-travel replay for Claude Code and Codex — deterministic offline agent testing
Relay agent for bridging Confident AI evaluation server to internal AI connections endpoints
Benchmark full-context-in-prompt vs. RAG for a document, against Ollama, vLLM, LiteLLM, Open WebUI, or any OpenAI-compatible backend
Claude Code Personal Analytics — a private, local, read-only analytics dashboard for your Claude Code sessions: costs, tokens, tools, skills, coding patterns, and wasted effort.
Deprecated: renamed to lanetrace. This package installs lanetrace and keeps `import agenttrace` and the `agenttrace` command working.
Tarification, comptage de tokens et affichage LLM partagés du parc
Lightweight SDK to track AI agent tool calls, costs, and latency.
Lightweight SDK to track AI agent tool calls, costs, and latency.
Evaluation-Driven Development (EDD) tools and helper libraries for AI Agent development
AgentLab — a modular framework for building, evaluating, and comparing LLM-powered agents.
LLM evaluation framework for developers
Composable RAG evaluation library for experiment-based testing
AgenticOps stress-search harness: deterministic GA/MC over modular simulations, BYOW adapters, replay JSON, and FEL evidence semantics.
Python library to support LLM testing.
Run a prompt through a LangChain (system + human) chain over Gemini on Vertex AI; optional LangSmith tracing.
Run a prompt through a LangChain (system + human) chain over Gemini on Vertex AI; optional LangSmith tracing.
Run a prompt through a LangChain (system + human) chain over Gemini on Vertex AI; optional LangSmith tracing.
Run a prompt through a LangChain (system + human) chain over Gemini on Vertex AI; optional LangSmith tracing.
Proxy Gemini (Vertex AI) completions wrapped in OpenTelemetry trace spans; returns the answer plus the trace/span id.
X-ray for LangGraph retrieval: one callback handler captures every retriever call, score, and relational path — rendered as an interactive graph.
Log every LLM API call with full request/response details for Hermes Agent
Production-grade, non-breaking live debugger and telemetry agent for Python.
Python SDK for AgentOps observability.
TTTPS Proof-of-Time success callback for LiteLLM: cryptographic audit-trail timestamps for LLM completions via the public KPP Provenance API
Benchmark llama.cpp models across candidate configurations
TTTPS Proof-of-Time callback handler for LlamaIndex: cryptographic audit-trail timestamps for LLM events via the public KPP Provenance API
Benchmark the LLM inference capacity of a server (llama-benchy orchestrator with auto-detection, real-workload sizing, charts and reports).
A dev tool for manually driving llm-agents-from-scratch's SupervisedTaskHandler one call at a time, over HTTP, via a React frontend.
Tamper-evident audit trails for LLM lifecycles: training, deployment, and monitoring.
OpenRouter LLM benchmark for repair-agent and validator-agent model selection.
Multi-format LLM API compliance testing tool — validates endpoints against OpenAI Chat, Open Responses, Anthropic, and Google GenAI specs.
A lightweight Prometheus exporter for LLM eval metrics: faithfulness, semantic drift, cost, and CI/CD regression gating.
The htop for LLM inference. Measured. Not guessed.
A multi-provider LLM-as-a-Judge consensus panel with chained criteria for concept matching validation.
Self-hosted LLM observability SDK for Python, LangChain, and LangGraph
Run-scoped LLM observability control plane backed by LiteLLM
The pre-merge check for LLM model, prompt, schema, and provider changes
A GitHub Action and CLI that detects risky changes to LLM prompts and AI configuration before they ship to production
Flight recorder for LLM agents: trace every call, replay runs deterministically, diff runs to find where behavior diverged.
Count distinct LLM users and find top token consumers across services, days and tools. Fast, small, portable summaries in Go and Python.
Local smoke tests for live LLM models, latency, and cost
Track LLM token usage, enforce policies & budget limits, and trigger alerts — across OpenAI, Groq, OpenRouter, AWS Bedrock, and custom providers.
Per-query token ledger for LLM pipelines: record what the API reports, answer with one GROUP BY. Zero dependencies, dashboard included.
Local LLM token efficiency middleware — zero cloud, zero tracking, coaches developers on token waste in the terminal
Track LLM API costs, tokens, and latency to MySQL
Lightweight terminal tool to check remaining credits / quota across LLM providers. Zero dependencies.
A general-purpose LLM verification framework: fine-grained reward + Pivot Preference Tournament best-of-N selection.
LLMIntent suite: Anatomy of any weighted LLM, fly-connectome prior, trajectory, MisAlign Flag, isolates, IV, latent inspect
Python library to support LLM testing.
LLM Runner Control is a CLI tool to interact with local and remote LLM, define multiple scenario variations (model × prompt × options), run it over a dataset and capture structured results for comparison or to build knowledge through meta analysis.
Profile LLM gateway logs for prefix-cache reuse potential, and export anonymized replayable traces
Passive LLM endpoint identity and stability instrumentation
MOSAIC: evaluating the Moral, Social and Individual dimensions of LLMs
CI-integrated evaluation harness for LLM agents
Agent 观测 SDK:模块四段(输入/结构/流程/输出)申报 + LangChain 自动采集 + HTTP 上报
Open-source CLI framework for evaluating RAG systems and AI Agents
A self-metering guardrail that counts and caps your OpenAI token usage against a daily (UTC) cap.
A vendor-neutral FinOps and measurement toolkit for AI usage, cost, efficiency, and business outcomes.
Lightweight LLM-as-judge evaluation CLI — score model outputs on 5 dimensions and compare runs (OpenAI / OpenRouter / Ollama)
OpenTelemetry Anthropic instrumentation
OpenTelemetry Official Langchain instrumentation
OpenTelemetry Official OpenAI instrumentation
OpenTelemetry Official OpenAI Agents instrumentation
Orqis SDK - one line to stream AI agent traces to Orqis self-healing ops
git diff, but for LLM behavior — behavioral regression testing for language models.
Fast local web viewer for pydantic-ai traces (list[ModelMessage] JSON dumps)
Large Language Model Instinct Testing under Survival - benchmark LLMs in a hand-built Crafter survival world.
Never ship unverified LLM output: multi-model councils, severity-graded audit gates, bounded retries, deterministic fallbacks. Zero dependencies.
Audit toolkit for retrieval, evaluation, benchmark, and security checks in RAG systems
A debugger for RAG applications: diagnoses retrieval, context, and grounding failures.
Lightweight JSONL tracing for debugging failed RAG answers.
A from-scratch CLI evaluation framework for RAG pipelines.
A pip-installable Python library providing drop-in RAG evaluation capabilities using RAGAS
A thin, CI-gated evaluation gate for RAG & LLM systems: golden set, LLM-judge (or heuristic) scorers, and band-based quality gates.