Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 17 of 57
Privacy first toolkit for analyzing your ChatGPT and Claude exports. Model tracking, prompt effectiveness analysis, cost efficiency metrics (POD/DLOD/POE), topic classification, and industry benchmarks. 49 tests. pip installable. Everything runs locally.
Hands-on LLM evaluation learning repo — local models via Ollama, no paid APIs, no maths. Covers deterministic eval, LLM-as-a-Judge, hallucination testing, prompt injection, RAG evaluation, and agent trajectory scoring.
Claude Code usage tracker and local dashboard for Pro/Max subscription limits - weekly cost-per-percent trend, quota forecasts, threshold alerts. ccusage-compatible.
Demo walkthrough and sample code for standardized multi-agent observability across clouds using Azure AI Foundry and OpenTelemetry GenAI semantic conventions. From Microsoft Build 2026.
Lightweight Python SDK for LLM inference logging and observability
See how many tokens your AI coding agents really burned.
The official Node.js SDK for Known Agents: AI agent analytics (bot traffic), LLM referral tracking, automatic robots.txt
Record, replay, and diff every LLM call your AI agent makes
A chunk-source-agnostic evaluation harness for RAG chunking strategies — measure recall, precision, and cost before picking one.
Reverse-engineer the RL reward an LLM was trained on — from black-box agentic coding behavior alone. Tested on Opus 4.8, Fable 5, GPT-5.5.
Open-source ESP32 dashboard for monitoring Claude Code and Codex usage, limits, and status in real time.
docker region:us
gradio region:us
gradio leaderboard region:us
gradio leaderboard region:us
gradio region:us
static region:us
gradio region:us
Cut LLM token usage on command output and API traffic — compress what enters your coding agent's context, meter what your apps spend.
Stack traces that tell the tale — a Logback appender that turns Java errors into AI-ready reports
Drop-in observability for the OpenAI Node SDK. Wrap your client once and every chat.completions.create and responses.create call lands in Voight: prompts, tokens, cache reads, tool calls, latency, errors.
SkillPortrait — skill-centered utility and safety evaluation for agent skill packages
Retroactive Claude Code Skill usage analytics. Single-file Python CLI, reads your existing ~/.claude transcripts, counts real Skill tool_use invocations (not prose mentions). No hook to install ahead of time.
Inspect Claude Code / Codex CLI conversations locally. Token breakdown, per-turn costs, compaction timeline, framework-overhead audit. Python + tiktoken.
Study many. Build your own. A CLI workbench for evaluating external frameworks and absorbing what proves out into your own systems — with evidence and decisions on record.
Verifiable agent actions. Every action becomes signed context for the next. Ed25519 signatures, Merkle log proofs, independently verifiable by anyone.
Check in detail your usage of Codex, with charts, price estimate and more !
Turn-level wake events for AI agents in herdr panes — pi extension, wait CLI, crash bridge, and Discworld callsigns for your workers
Rust superset of httpbin.org: 84 drop-in endpoints, mock OpenAI/Anthropic LLM APIs, and a full OAuth 2.0 server - for testing HTTP clients, proxies, and AI agents.
LLM evaluation for Java — LLM-as-judge, RAG, agent-trajectory and structured-output evaluators for Spring AI and LangChain4j. A Java port of LangChain's OpenEvals.
Role-aware, permission-enforced benchmark for LLM agents that operate HPC systems — SLURM, telemetry, RBAC, docs, facility. A policy violation hard-fails the task. 88 tasks (10 categories x 5 roles), 29 reproducible environments with 6 rebuilt from real Marconi100 ExaData, scored on 7 weighted dimensions.
Animated 3D virtual office for Hermes AgentOS subagents — real-time isometric AI visualization with Three.js.
See what Claude Code and Codex actually did. Local-first observability for every session: which tools ran, what each sub-agent was told, which turn blew the budget. Cost, tokens, error alerts, full-text search. SQLite on your machine, nothing leaves it.
Observability infrastructure for self-improving agents: every run written into the repository as plain files the agent can read
A visual analytics platform for engineering, executing, and evaluating reproducible agentic AI teams using experimental design methods.
How do you KNOW if a 'skill' is any good - deterministically? This runs controlled skill-vs-no-skill experiments. It doesn't just ask, “Did the skill work?” It asks “Did we actually give the skill to the model, did the comparison remain fair, what changed, what did it cost, and do we have enough evidence to say that confidently?”
Microsoft Foundry Portal에서 직접 해보는 Evaluation Engineering
Code for "When Agents Slow Down: Understanding LLM Agents' Test-Time Strategies via Elo-per-token Analysis" (arXiv:2609.15309) — paper: https://arxiv.org/abs/2609.15309
Local-first observability and command center for AI agents.
Deterministic flight recorder, $0 CI replay engine, and time-travel debugger for AI agents.
Interactive LLM prompt-caching calculator with editable rates, savings and break-even analysis.
Codex usage dashboard with weekly quotas, credit balances and history, and API-equivalent cost estimates. Run locally or sync multiple machines privately.
Governance for AI agent calls: check entitlements before a call, meter usage after. Twelve providers and gateways, none of them a dependency.
docker autotrain region:us
Open-source instrumentation for AI coding agents. Capture activity, concurrency and working patterns across Claude Code, Codex, Cursor and more.
A runtime learning loop for coding agents. Turns Claude Code, Codex, Cursor, Antigravity and model-router runs into human-approved, measured changes to the agent harness: guard rules, skills, sub-agents and routes. It tunes the harness, not the model.
Unofficial Go SDK for Langfuse - trace, monitor, and evaluate your LLM applications with async batch processing and specialized observation types
A professional Python framework for testing prompt strength, quality, and real autonomous-agent effectiveness.
Native macOS command center for OpenClaw AI agents. Real-time dashboard, live chat, service monitoring. Built with SwiftUI.
Open-source replay and regression testing for AI agents.
Your behavior is the signal. Not your words. — Behavioral intelligence for AI agents, built into your MacBook notch.
Track what your AI agents cost including files, commits, tokens, budgets. CLI , Dashboard+ API. Works with OpenClaw, Claude Code, and any agent framework.
The predictable, observable agent harness.
AgentTrace is an open-source, local-first step debugger for AI agents. It provides a Python SDK for tracing your agent runs and a web UI to inspect spans, tool calls, prompts, and responses as an interactive tree.
Tool Confirmation Lock with Bayesian Regret
A forensic analysis of how local AI IDEs and wrappers secretly inject 'CHECKPOINT' mechanisms and override prompts to lobotomize your LLMs (Claude/Gemini) for token-saving. It's not the model degrading; it's the infrastructure hijacking your context.
A session trace format for capturing human-AI coding interactions across frameworks.
RAG Evaluation Pipeline with RAGAS
CoAgent - Control software to log, test, and monitor generative ai applications and agents.
Simple and blazingly fast dynamic evaluation platform for benchmarking Large Language Models
This project helps you check your JetBrains AI Assistant quota usage information. 本项目帮助您查看 JetBrains AI Assistant 的配额使用情况。
OpenAI Cost Calculator is a simple tool, implemented in Python and JavaScript, for estimating the cost of using OpenAI's GPT-3 or GPT4 API.
Thread-Safe OpenAI Cost Logger
macOS menu bar app + desktop widget showing Claude usage in real time — session, weekly, Sonnet 4.5 & extra credits. One-command install, no Apple Developer account needed.
A tool for evaluating and tracking your RAG experiments. This repo contains the Python SDK for logging to Tonic Validate.
Test, compare, and optimize your AI prompts in minutes
Learn to build trustworthy AI with systematic evaluations in Azure AI Foundry. The session covers quality, safety, agent and custom evaluators
Enterprise LLM Evaluation & Responsible AI Framework — Benchmark bias, hallucination, PII leakage, and toxicity across Healthcare, BFSI, Retail & Legal industries. Supports OpenAI, Anthropic, Gemini & HuggingFace. Python SDK + CLI + Web Dashboard. 191 tests. Compliance-ready reports.
Benchmark harness for A/B testing Claude Code plugins against OOLONG long-context reasoning tasks. Compare truncation vs RLM-RS recursive chunking strategies. Features Claude Code hooks integration, SQLite persistence, and comprehensive scoring aligned with the OOLONG paper methodology.
AgentOps control plane for AI agent execution. Rust policy plane, hash-chained audit, deny-by-default tool allowlists and budget enforcement, eval gating in CI. Single maintainer, moving fast, read the README before you deploy it. Docs: https://sattyamjjain.github.io/ferrumdeck/
A simple web tool to test any OpenAI compatible API
Open-source AI + data cost intelligence — 18 connectors (Claude, GPT, Gemini, dbt, warehouses, BI, cloud, CI/CD), cache-tier visibility, anomaly detection, MIT licensed
An eval harness and report on some of the latest LLMs that can be run locally
Benchmark test repo: Next.js project with intentional product-context gotchas for AI agent evaluation
Unified benchmark for evaluating architectural design choices in LLM-based single-agent and multi-agent frameworks across orchestration, memory, planning, specialization, and coordination.
Benchmark, observe, and govern tool-using AI agents before production.
Observability for AI agents and LLM-powered applications. With per-user cost attribution, plan tracking, and optional Solana-anchored audit.
Self-healing observability for AI agents. Discover, monitor, and auto-recover multi-agent systems — without a cloud dependency.
Drop-in observability for the Anthropic Node SDK. Wrap your client once and every messages.create call lands in Voight — prompts, tokens, cache reads, cache creations, tool use, latency, errors.
The missing observability layer for AI agents. Real-time session visualization. Every prompt. Every tool call. Every model switch. Live. Drop-in OpenAI/Anthropic gateway. Self-hosted. Bun + SQLite.
gradio region:us
streamlit region:us
gradio region:us
streamlit region:us
static region:us
The open-source benchmark for LLM memory decay. Measure how Naive, RAG, Chunked RAG, Cascading, and SummaryMemory degrade over 100 conversation turns. Ebbinghaus forgetting curves, 5-provider LLM eval, multi-seed CI. No API key needed.
Kotlin Multiplatform semantic cache for LLM calls: eleven guards against false cache hits, measured on blind corpora, coroutine-first.
Open-source LLM cost monitoring & optimization platform. Track spend across OpenAI, Anthropic, DeepSeek & OpenRouter in real-time.
See where your Claude Code tokens actually go. Profiles the session logs Claude Code already writes, scores each project's context hygiene 0–100, and generates fixes backed by your own measured numbers. Local, zero deps, no telemetry.
The agentic council is a fully configurable wrapper to introduce a novel method for cross-family critical red-teaming and evaluation of a "leader" agent's work. This is NOT an oracle. It is a toolchain designed to force agents into more honest, thoroughly verified workflows in a configurable, agent-agnostic way.
Lightweight AI API cost tracking proxy - one-line base_url change, transparent token and cost monitoring.
Local debugger for AI agents. Answer the question: "Why did my agent behave this way?" with visual replay, behavior diff, and full visibility into prompts, context, memory, retrieval, and tool calls. Open source. Works with Langchain.
Codex额度悬浮窗:一眼查看 Codex 本周剩余额度 · Native float widget for weekly Codex quota
Synthetic monitoring and CI smoke tests for LLM inference endpoints.
Get notifications about leaks that might be bleeding your tokens. Based on real-time analysis on your coding agent sessions.
OpenClaw skill: genpark foil
GenPark AI Agent Skill - Pairwise Bradley-Terry and Elo rating tournament engine for ranking multi-agent strategies, prompt mutations, and generated tool solutions.
GenPark AI Agent Skill - Detects LLM hallucinations and confabulations by clustering sampled responses into semantic equivalence classes and computing semantic entropy.
GenPark AI Agent Skill - Distribution-free conformal prediction sets with rigorous marginal coverage guarantees, calibrated non-conformity scores, and split conformal inference.