Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 46 of 57

gosce-traced-llm-proxy

Proxy Gemini (Vertex AI) completions wrapped in OpenTelemetry trace spans; returns the answer plus the trace/span id.

imbue-mngr-claude-usage

Claude usage data provider for `mngr usage` (per-agent statusline shim that captures rate-limits + cost + session_id)

rag-audit

CLI + library to audit and benchmark RAG pipelines

ragcheck-cli

Lighthouse for RAG systems - diagnose and fix your retrieval pipeline

@agenttrace-io/auto-instrument

Zero-code auto-instrumentation for AI agents. Detects and traces LangChain, CrewAI, AutoGen, OpenAI, Anthropic, and raw HTTP calls automatically.

agentcassette

Deterministic agent test recorder and replayer. Record live runs, replay as mocks. Zero dependencies.

agenttrace-io

AgentTrace Python SDK - Drop-in tracing for any AI agent

llamaindex-ledgerproof

LedgerProof callback handler for LlamaIndex — Article 50 transparency receipts for RAG and agent workflows.

openai-ledgerproof

LedgerProof adapter for the OpenAI Python SDK — emits Article 50 transparency receipts as a side channel.

stillrunning-langchain

One-line monitoring for LangChain. A BaseCallbackHandler that pings StillRunning on every chain/agent run with duration, tokens, cost, model, and tool-call count.

runtime-monitor-sdk

TypeScript/JavaScript SDK for the AI Agent Runtime Monitor — cost guardrails, hallucination detection & audit trail

@tensflare/crewai

CrewAI middleware that injects Truss recording into crew task execution

agentlens-eval

Local-first observability + eval for AI agents: trace runs, detect loops, render an HTML report. No backend.

gosce-llm-observability-orchestration-agent-langchain

Run a prompt through a LangChain (system + human) chain over Gemini on Vertex AI; optional LangSmith tracing.

rankigi-langchain

LangChain (Python) callback handler that adds RANKIGI cryptographic receipts to every LLM, tool, agent, and retriever step.

rankigi-litellm

LiteLLM CustomLogger that adds RANKIGI cryptographic receipts to every model call routed through LiteLLM.

@tensflare/autogen

AutoGen middleware that injects Truss recording into multi-agent conversations

@isplay/adapter-langgraph

LangGraph node, model, stream, and tool instrumentation for isplay.

agent-event-protocol

Python SDK for the Agent Event Protocol (AEP) — structured observability for multi-agent AI systems

agent-tracker

Telemetry SDK for self-built AI trading agents.

agent-usage-manager

Local htop-style dashboard and guarded kill switch for headless AI-agent processes.

agenticmeter-cli

A linter for AI agent runs: find the bugs that waste money. Local-first, zero dependencies.

ai-agent-profiler

Runtime profiler and optimization engine for AI agents using MCP

claude-agent-cassette

Record & replay the claude-agent-sdk wire for deterministic, offline tests.

openai-cost-guard

Cost tracking and budget enforcement for Azure OpenAI API calls

autoevals

Universal library for evaluating AI models

@agent-ils/logger

Local JSONL logger for AI-assisted debugging, with browser writes and CLI reads.

@jenz-ai/openai-agents

Zero-config OpenAI Agents SDK adapter for Jenz observability.

sa-sdk-javascript

official sensorsdata javascript sdk

agentsynth-ai

Outcome-verified agent trajectories, benchmarks, and RL environments — with a live leaderboard and a CI gate. Offline-first.

agenttrace-runtime

Open-source runtime tracing and diagnostics for AI agent execution flows.

budget-guard-agents

Per-task budget, loop detection, and kill-switch middleware for agent LLM calls. v0, deterministic and dependency-free.

langchain-nebeso

LangChain callback handler for NeBeso AI cost tracking

llama-index-callbacks-nebeso

LlamaIndex callback handler for NeBeso AI cost tracking

llm-canary

Regression canary for LLM apps: YAML test suites, offline-first assertions, baseline diffing, and agent-trace policy gates for CI

llm-eval-unb

Framework open-source para avaliação de confiabilidade de chatbots baseados em LLMs

openai-agents-valta

OpenAI Agents SDK tool for hard spending limits on AI agents via Valta

@tarko/agio

Agio (Agent Insights and Observations) - A standardized multimodal AI Agent Server monitoring protocol for server-side agent operation monitoring and analytics

agent-skill-eval

Framework for evaluating agent skills across OpenCode, Claude Code, and Codex

agent-trace-eval

Golden trace regression evaluation for tool-using AI agents

agentburn

Which 5-hour window took you out, and where the money goes. Local usage/cost profiler for Claude Code, OpenClaw and Hermes Agent. Zero dependencies, nothing leaves your machine.

spool-agent

Spool SDK for Python — trace LLM agent runs into the local Spool store

trovis-agents

Connect your agents to Trovis — two lines of code for full observability

@jnmetacode/tracelet

Local-first DevTools for AI agents. See every tool call, prompt, and token — 100% on your machine. Zero cloud, zero signup, one command.

sentinel-agentos

Sentinel AgentOS — 确定性 Guard 层 + 分层记忆 + 自动评估,让任何 Agent 变得可靠、可审计、可改进

agentrec

Replay recorded LLM API traffic against a candidate model: cross-provider migration & regression reports (OpenAI/Anthropic/Gemini/Mistral) with scoring, cost and latency, built on httpx record/replay

gosce-llm-orchestration-traced-agent

Run a prompt through a LangChain (system + human) chain over Gemini on Vertex AI; optional LangSmith tracing.

groundcheck-rag

A small, cheap hallucination / grounding detector — does an AI answer stay supported by its source?

respan-instrumentation-aws-bedrock

Respan instrumentation plugin for AWS Bedrock Runtime

respan-instrumentation-cohere

Respan instrumentation plugin for Cohere

respan-instrumentation-litellm

Respan instrumentation plugin for LiteLLM

tokensaver-agent

Open-source runtime ROI diagnosis toolkit for AI Agent applications.

@tonyclaw/llm-inspector

Local LLM API inspector for AI coding tools, with real-time request and response capture.

@kontourai/flow

Process transparency for agentic work.

agentcanvas

Visualize Pydantic AI agent workflows from Logfire traces as an interactive HTML diagram.

agentrel

Reliability and reproducibility statistics for stochastic agent / tool-use evals.

agentsproof

Observability and proof reporting for AI agents

llm-evaluation-toolkit

A lightweight toolkit for evaluating LLM outputs: BLEU, ROUGE, semantic similarity, and LLM-as-a-Judge

llm-price-tracker

Fetch, normalise, snapshot, and diff official LLM API token pricing.

lowresource-llm-evaluation

Métricas para evaluar LLMs en su capacidad de hablar una lengua sin datasets de Q&A

rag-agent-eval-ci

CI evaluation framework for RAG and AI agents: groundedness, retrieval quality, hallucination, citations, latency, cost, and regression gating before production.

ragas-ainative

Configure ragas RAG evaluation to use AINative's free LLMs + embeddings — no OpenAI key needed.

respan-instrumentation-groq

Respan instrumentation plugin for Groq

respan-instrumentation-mistralai

Respan instrumentation plugin for Mistral AI

respan-instrumentation-ollama

Respan instrumentation plugin for Ollama

useagentledger

AgentLedger Python SDK — track, attribute, and govern AI agent spend across OpenAI, Anthropic, and Gemini.

cassette-sdk

Record/replay LLM & agent API calls for fast, free, deterministic tests. In-process recorder + optional gateway shim.

cloptima-llm-observability

LLM usage telemetry SDK for Cloptima reporting, attribution, and analytics

llm-judge-kit

Provider-agnostic, reproducible, typed LLM-as-a-judge — a small primitive you can depend on.

loupe-llm

Python SDK for Loupe — observability for LLM apps (latency, tokens, cost, errors).

xone-agent-failure-packet

Create redacted, shareable debug packets from failed AI agent runs.

@checklabs/openai

OpenAI agent adapter and LLM-judge backend for CheckAI.

@checkai-dev/openai

OpenAI agent adapter and LLM-judge backend for CheckAI.

@tensflare/langgraph

LangGraph middleware that injects Truss recording into agent loop steps

fableforge-agent-profiler

Profile and classify agent behavior patterns from traces

fableforge-agent-telemetry

Datadog for AI agents — real-time observability, token tracking, cost estimation, error rates

fableforge-bench-agent

HumanEval for tool use — a standardized benchmark for evaluating LLM tool-use capabilities

rageval-core

RAG evaluation with evidence — see which sentence hallucinated, not just a score

sentrinode-llm

Drop-in observability for LLM calls — cost, tokens, latency, errors via OpenTelemetry GenAI conventions.

AI Agent Token Cost Calculator

This tool helps you figure out your monthly AI agent expenses. It shows you how much money you might be wasting and suggests simple ways to lower your costs by

agentlogger

Local-first AI agent observability — auto-trace LLM calls and tools, view runs in a local dashboard

@expressots/studio-agent

Instrumentation agent for ExpressoTS Studio - route discovery, tracing, and request recording

@raindrop-ai/opencode-plugin

Raindrop observability plugin for OpenCode — automatic session/event/span tracing

artemisllmbench

Artemis LLM Benchmark — correctness validation and performance benchmarking for any OpenAI-compatible LLM serving endpoint

gauntlet-agent

Break your agent before your users do, and audit RL environments for reward hacks. Adversarial stress-testing, regression suites and environment audits.

llmprof

pprof for your LLM context - see where every token and dollar goes.

ollama-arena

Pair-wise ELO evaluation arena for local LLMs.

prompt-analytics-for-claude-code

Prompt-level analytics for Claude Code — tokens, costs, and session insights from your local JSONL files

rag-eval-gate

CI/CD-integrated RAG evaluation pipeline — quality gate for AI chatbots using Ragas + Groq LLM judge

ragobserve

RAGObserve: local-first observability, debugging and evaluation for RAG systems. The MLflow for RAG.

useagentwatch

AI agent observability SDK for agencies

versus-llm

Two LLMs enter. One verdict leaves.

@renaissancemind/tokenflow

Local token usage collector for multi-device AI agent accounting.

@surpradhan/aep

Node.js / TypeScript SDK for the Agent Event Protocol (AEP) — framework-neutral observability for multi-agent AI systems, with LangChain.js / LangGraph auto-instrumentation.

@tracegate/adapters

Framework adapters and observability exports for TraceGate.

@dawn-ai/evals

<p align="center"> <img src="https://raw.githubusercontent.com/cacheplane/dawnai/main/docs/brand/dawn-logo-horizontal-black-on-white.png" alt="Dawn" width="180" /> </p>

@dawn-ai/testing

<p align="center"> <img src="https://raw.githubusercontent.com/cacheplane/dawnai/main/docs/brand/dawn-logo-horizontal-black-on-white.png" alt="Dawn" width="180" /> </p>

reddial

Multi-agent simulation harness that stress-tests conversational AI agents with adversarial synthetic users, then grades every transcript with auditable decision-tree judges.

agentic-rubric-runner

A lightweight, auditable agent workflow for document-grounded PDF generation and rubric-based evaluation

claude-credit-audit

Forecast which agent calls burn Anthropic Claude Agent SDK credit after the 2026-06-15 billing change — Claude Code / claude-code-action / claude -p cost, monthly risk, and cheaper alternatives.

More in Agent frameworks & infrastructure