Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 46 of 57
Proxy Gemini (Vertex AI) completions wrapped in OpenTelemetry trace spans; returns the answer plus the trace/span id.
Claude usage data provider for `mngr usage` (per-agent statusline shim that captures rate-limits + cost + session_id)
CLI + library to audit and benchmark RAG pipelines
Lighthouse for RAG systems - diagnose and fix your retrieval pipeline
Zero-code auto-instrumentation for AI agents. Detects and traces LangChain, CrewAI, AutoGen, OpenAI, Anthropic, and raw HTTP calls automatically.
Deterministic agent test recorder and replayer. Record live runs, replay as mocks. Zero dependencies.
AgentTrace Python SDK - Drop-in tracing for any AI agent
LedgerProof callback handler for LlamaIndex — Article 50 transparency receipts for RAG and agent workflows.
LedgerProof adapter for the OpenAI Python SDK — emits Article 50 transparency receipts as a side channel.
One-line monitoring for LangChain. A BaseCallbackHandler that pings StillRunning on every chain/agent run with duration, tokens, cost, model, and tool-call count.
TypeScript/JavaScript SDK for the AI Agent Runtime Monitor — cost guardrails, hallucination detection & audit trail
CrewAI middleware that injects Truss recording into crew task execution
Local-first observability + eval for AI agents: trace runs, detect loops, render an HTML report. No backend.
Run a prompt through a LangChain (system + human) chain over Gemini on Vertex AI; optional LangSmith tracing.
LangChain (Python) callback handler that adds RANKIGI cryptographic receipts to every LLM, tool, agent, and retriever step.
LiteLLM CustomLogger that adds RANKIGI cryptographic receipts to every model call routed through LiteLLM.
AutoGen middleware that injects Truss recording into multi-agent conversations
LangGraph node, model, stream, and tool instrumentation for isplay.
Python SDK for the Agent Event Protocol (AEP) — structured observability for multi-agent AI systems
Telemetry SDK for self-built AI trading agents.
Local htop-style dashboard and guarded kill switch for headless AI-agent processes.
A linter for AI agent runs: find the bugs that waste money. Local-first, zero dependencies.
Runtime profiler and optimization engine for AI agents using MCP
Record & replay the claude-agent-sdk wire for deterministic, offline tests.
Cost tracking and budget enforcement for Azure OpenAI API calls
Universal library for evaluating AI models
Local JSONL logger for AI-assisted debugging, with browser writes and CLI reads.
Zero-config OpenAI Agents SDK adapter for Jenz observability.
official sensorsdata javascript sdk
Outcome-verified agent trajectories, benchmarks, and RL environments — with a live leaderboard and a CI gate. Offline-first.
Open-source runtime tracing and diagnostics for AI agent execution flows.
Per-task budget, loop detection, and kill-switch middleware for agent LLM calls. v0, deterministic and dependency-free.
LangChain callback handler for NeBeso AI cost tracking
LlamaIndex callback handler for NeBeso AI cost tracking
Regression canary for LLM apps: YAML test suites, offline-first assertions, baseline diffing, and agent-trace policy gates for CI
Framework open-source para avaliação de confiabilidade de chatbots baseados em LLMs
OpenAI Agents SDK tool for hard spending limits on AI agents via Valta
Agio (Agent Insights and Observations) - A standardized multimodal AI Agent Server monitoring protocol for server-side agent operation monitoring and analytics
Framework for evaluating agent skills across OpenCode, Claude Code, and Codex
Golden trace regression evaluation for tool-using AI agents
Which 5-hour window took you out, and where the money goes. Local usage/cost profiler for Claude Code, OpenClaw and Hermes Agent. Zero dependencies, nothing leaves your machine.
Spool SDK for Python — trace LLM agent runs into the local Spool store
Connect your agents to Trovis — two lines of code for full observability
Local-first DevTools for AI agents. See every tool call, prompt, and token — 100% on your machine. Zero cloud, zero signup, one command.
Sentinel AgentOS — 确定性 Guard 层 + 分层记忆 + 自动评估,让任何 Agent 变得可靠、可审计、可改进
Replay recorded LLM API traffic against a candidate model: cross-provider migration & regression reports (OpenAI/Anthropic/Gemini/Mistral) with scoring, cost and latency, built on httpx record/replay
Run a prompt through a LangChain (system + human) chain over Gemini on Vertex AI; optional LangSmith tracing.
A small, cheap hallucination / grounding detector — does an AI answer stay supported by its source?
Respan instrumentation plugin for AWS Bedrock Runtime
Respan instrumentation plugin for Cohere
Respan instrumentation plugin for LiteLLM
Open-source runtime ROI diagnosis toolkit for AI Agent applications.
Local LLM API inspector for AI coding tools, with real-time request and response capture.
Process transparency for agentic work.
Visualize Pydantic AI agent workflows from Logfire traces as an interactive HTML diagram.
Reliability and reproducibility statistics for stochastic agent / tool-use evals.
Observability and proof reporting for AI agents
A lightweight toolkit for evaluating LLM outputs: BLEU, ROUGE, semantic similarity, and LLM-as-a-Judge
Fetch, normalise, snapshot, and diff official LLM API token pricing.
Métricas para evaluar LLMs en su capacidad de hablar una lengua sin datasets de Q&A
CI evaluation framework for RAG and AI agents: groundedness, retrieval quality, hallucination, citations, latency, cost, and regression gating before production.
Configure ragas RAG evaluation to use AINative's free LLMs + embeddings — no OpenAI key needed.
Respan instrumentation plugin for Groq
Respan instrumentation plugin for Mistral AI
Respan instrumentation plugin for Ollama
AgentLedger Python SDK — track, attribute, and govern AI agent spend across OpenAI, Anthropic, and Gemini.
Record/replay LLM & agent API calls for fast, free, deterministic tests. In-process recorder + optional gateway shim.
LLM usage telemetry SDK for Cloptima reporting, attribution, and analytics
Provider-agnostic, reproducible, typed LLM-as-a-judge — a small primitive you can depend on.
Python SDK for Loupe — observability for LLM apps (latency, tokens, cost, errors).
Create redacted, shareable debug packets from failed AI agent runs.
OpenAI agent adapter and LLM-judge backend for CheckAI.
OpenAI agent adapter and LLM-judge backend for CheckAI.
LangGraph middleware that injects Truss recording into agent loop steps
Profile and classify agent behavior patterns from traces
Datadog for AI agents — real-time observability, token tracking, cost estimation, error rates
HumanEval for tool use — a standardized benchmark for evaluating LLM tool-use capabilities
RAG evaluation with evidence — see which sentence hallucinated, not just a score
Drop-in observability for LLM calls — cost, tokens, latency, errors via OpenTelemetry GenAI conventions.
This tool helps you figure out your monthly AI agent expenses. It shows you how much money you might be wasting and suggests simple ways to lower your costs by
Local-first AI agent observability — auto-trace LLM calls and tools, view runs in a local dashboard
Instrumentation agent for ExpressoTS Studio - route discovery, tracing, and request recording
Raindrop observability plugin for OpenCode — automatic session/event/span tracing
Artemis LLM Benchmark — correctness validation and performance benchmarking for any OpenAI-compatible LLM serving endpoint
Break your agent before your users do, and audit RL environments for reward hacks. Adversarial stress-testing, regression suites and environment audits.
pprof for your LLM context - see where every token and dollar goes.
Pair-wise ELO evaluation arena for local LLMs.
Prompt-level analytics for Claude Code — tokens, costs, and session insights from your local JSONL files
CI/CD-integrated RAG evaluation pipeline — quality gate for AI chatbots using Ragas + Groq LLM judge
RAGObserve: local-first observability, debugging and evaluation for RAG systems. The MLflow for RAG.
AI agent observability SDK for agencies
Two LLMs enter. One verdict leaves.
Local token usage collector for multi-device AI agent accounting.
Node.js / TypeScript SDK for the Agent Event Protocol (AEP) — framework-neutral observability for multi-agent AI systems, with LangChain.js / LangGraph auto-instrumentation.
Framework adapters and observability exports for TraceGate.
<p align="center"> <img src="https://raw.githubusercontent.com/cacheplane/dawnai/main/docs/brand/dawn-logo-horizontal-black-on-white.png" alt="Dawn" width="180" /> </p>
<p align="center"> <img src="https://raw.githubusercontent.com/cacheplane/dawnai/main/docs/brand/dawn-logo-horizontal-black-on-white.png" alt="Dawn" width="180" /> </p>
Multi-agent simulation harness that stress-tests conversational AI agents with adversarial synthetic users, then grades every transcript with auditable decision-tree judges.
A lightweight, auditable agent workflow for document-grounded PDF generation and rubric-based evaluation
Forecast which agent calls burn Anthropic Claude Agent SDK credit after the 2026-06-15 billing change — Claude Code / claude-code-action / claude -p cost, monthly risk, and cheaper alternatives.