Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 9 of 57

SkillLens★ 45

Open-source self-hosted web tool for evaluating Agent Skills with rubric scores, Deep Review, and improvement suggestions.

ignite25-PREL13-observe-manage-and-scale-agentic-ai-apps-with-microsoft-foundry★ 45

Learn How To Observe, Manage, and Scale, Agentic AI Apps Using Azure AI Foundry - with this hands-on workshop

langeval★ 45

Evaluation for AI apps and agent

llmBench★ 45

llmBench is a high-depth benchmarking tool designed to measure the raw performance of local LLM runtimes (Ollama, llama.cpp) while providing deep hardware intelligence.

rag-evaluator★ 45

A library for evaluating Retrieval-Augmented Generation (RAG) systems (The traditional ways).

PersonaGym★ 45

[EMNLP 2025] Dataset and Code of "PersonaGym: Evaluating Persona Agents and LLMs"

sideseat★ 45

Unified workbench for building and debugging AI Agents

leaderboard★ 45

PinchBench is a benchmarking system for evaluating LLM models as OpenClaw coding agents. Made with 🦀 by the humans at https://kilo.ai

burn0★ 45

Lightweight cost observability for every API call in your stack. Track LLM tokens, database ops, and external services from your terminal.

clawlens★ 45

Agent observability and guardrails for OpenClaw — risk scoring, audit trails, dashboard.

llm_cost_tracker★ 45

LLM spend tracking and budgets for Rails — by user, feature, or any tag, in your own database, no proxy.

bench-loop★ 45

Local-first CLI for benchmarking LLMs on real hardware — quality, speed, reliability, and a real multi-turn agent loop.

flowcept★ 45

Runtime provenance for AI and scientific workflows—capture, enrich, and query workflow data via observability adapters and code annotation across edge, cloud, and HPC.

CoDA-Bench★ 45

CoDA-Bench is a benchmark for code agents on data-intensive tasks. 🎈代码智能体能搞定数据密集型任务吗?

ToolBudgetBench★ 45

Benchmark and demo harness for measuring LLM agent tool-context budget tradeoffs.

agent-trail★ 45

Local dashboard for Claude Code, Codex, OpenCode, OpenClaw and Qoder session history, token usage, cost tracking, tool calls and subagent replay.

tokenmeter★ 45

htop for AI Agents — monitor token usage, costs & tool calls across Claude Code and Codex in real time

tastebench★ 45

Taste-Bench: measuring the long-horizon judgment of LLM agents at real decision forks

ai-beacon★ 45

A web dashboard for monitoring and orchestrating AI coding agents across your devices.

lowlevel-toolkit★ 44

Ruby gem for calling observability APIs

fluent-plugin-multiprocess★ 44

Multiprocess agent plugin for Fluentd

geval★ 44

Decision orchestration and reconciliation for AI changes.

java-agent★ 44

Agent-based OpenTracing instrumentation in Java

ml-dev-bench★ 44

ML-Dev-Bench is a benchmark for evaluating AI agents against various ML development tasks.

n8n-nodes-openai-langfuse★ 44

An n8n community node that brings Langfuse observability to your OpenAI chat workflows.

APTBench★ 44

Code for "APTBench: Benchmarking Agentic Potential of Base LLMs During Pre-Training"

clawd-insights★ 44

Your agents do the work — you keep the knowledge. A local-first dashboard (+ desktop pet) that records and replays your Claude Code, Codex CLI and Cursor sessions: timeline, AI session summaries, analytics — all on your machine.

skillbench★ 44

Adversarial, deterministic benchmarking for AI agent skills — measures whether a skill makes the agent measurably better. Includes biomedical eval cases.

ChipCompAgent★ 44

This is an AI agent to compare the baselines of different type of AI chips. It simplifies the work of AI product managers.

dsh-trace-compare★ 44

Trace Compare & Live Maze for DeepSeek Harness: visualize agent exploration (main path, detours, backtracks) from session logs or live sessions

reactive-agents★ 43

Create and improve AI agents that get better over time with automatic optimization and continuous learning

QwenClawBench★ 43

General Agent Benchmark for OpenClaw, made by Qwen Team, Alibaba Group.

apm-agent-android★ 43

Elastic APM Android Agent

diRAGnosis★ 43

Diagnose the performance of your RAG🩺

pgg_bench★ 43

Public Goods Game (PGG) Benchmark: Contribute & Punish is a multi-agent benchmark that tests cooperative and self-interested strategies among Large Language Models (LLMs) in a resource-sharing economic scenario. Our experiment extends the classic PGG with a punishment phase, allowing players to penalize free-riders or retaliate against others.

eval-engineer★ 43

Bring Galileo powered eval workflows into Claude Code and Codex.

tokcat★ 43

Native Swift AI token usage monitor for the macOS menu bar — track Claude Code, Codex, Cursor, Copilot and other AI coding agent usage with a cute 2D/3D contribution graph.

MASEval★ 43

Multi-Agent LLM Evaluation Docs: https://maseval.readthedocs.io/

AutoSciRub★ 43

Learning to Evaluate Before Improving: Automatic Rubric Induction for Automatic Research Agents

TokenStep★ 43

Native macOS menu bar app that tracks local AI coding agent token usage like daily steps.

unbox-ai★ 43

Visualize AI context & traces.

RubricLoop★ 43

Rubric-driven evaluation for AI agents. Executable criteria, code and model judges, and inspectable trace reports.

TokenTracker★ 42

Drop-in LLM cost tracker — change one import line, see where your money goes. Supports OpenAI, OpenRouter, Azure, Ollama.

convelyze★ 42

Visualize your ChatGPT usage with interactive charts and insights.

jeeves★ 42

AI agent conversation history browser

2024-openai-gpt-hiring-racial-discrimination★ 42

Data and materials to reproduce Bloomberg's investigation into racial and gender bias in OpenAI's GPT

opsml★ 42

AI lifecycle platform for classical ML and agentic systems. Versioned, encrypted registries for data, models, experiments, prompts, agents, MCPs, and skills. Pre/deploy/post-deployment patterns. Real-time monitoring and evaluation via Scouter. Rust core, governed by design

argos★ 42

An agentic system for autonomously generating explainable and reproducible time-series anomaly detection rules using LLMs.

PlanBench-XL★ 42

Official Repository for our paper: PlanBench-XL: Evaluating Long-Horizon Planning of LLM Tool-Use Agents in Large-Scale Tool Ecosystems

agentpeek★ 42

Peek into your agents. Real-time visualization for multi-agent AI systems.

Agent-EvalKit★ 42

AI-driven toolkit that automates evaluation processes for AI agents

gpt-4o-mini_battles★ 42

gradio region:us

ARIES★ 42

ARIES is an open-source experimentation framework for agent serving systems research.

Syrtis-Windows★ 42

Windows port of TokenBar — a tray monitor for AI coding-agent token usage, cost, and subscription quota. WinUI 3 shell on the same Rust parsing core.

claude-notch★ 41

Track your Claude API usage from your MacBook's notch

FlowMetr★ 41

Workflow, Pipeline & AI Agent Observability Platform. Metrics, logs and traces for all your automations.

clawtrol★ 41

Mission control for your AI agents 🎮

bioagent-bench★ 41

Benchmark for evaluating LLM agents in bioinformatics

mcp-eval★ 41

Lightweight eval framework for MCP servers, built on mcp-agent

claude-notch-tracker★ 41

Live Claude, Codex, Antigravity, DeepSeek and opencode-go usage limits in your Mac's notch.

ko-chatbot-arena-leaderboard★ 41

static region:us

ToolBudgetBench★ 41

Benchmark and demo harness for measuring LLM agent tool-context budget tradeoffs.

CoffeeBench★ 41

[NeurIPS 2026] CoffeeBench: Benchmarking Long-Horizon LLM Agents in Heterogeneous Multi-Agent Economies

opentelemetry-hooks★ 41

OpenTelemetry integration for AI Agents Cursor / Copilot / Gemini / Claude / Codex that records agent activity and exports detailed telemetry with OTLP protocol

NanoHarness★ 41

Compact AI agent runtime control plane with governed tools, HITL approval, resumable execution, trace-driven evaluation, and SWE-bench-shaped evidence.

codex-observability-plugin★ 41

Trace OpenAI Codex CLI coding sessions, agent turns, tool calls, token usage, and subagents with Langfuse for comprehensive observability.

ecotrace★ 41

Lightweight, process-scoped energy and carbon instrumentation for Python applications, AI models, and LLM pipelines.

june-2025-coding-agent-report★ 40

Comprehensive evaluation of 15 AI coding agents (Cursor, Copilot, Claude, Replit, v0, Warp, etc.) with implementations, screenshots, and professional scoring. Published on Turing Post.

function-calling-eval★ 40

A framework for evaluating function calls made by LLMs

agentune★ 40

Tune your AI Agent to best meet its KPI with a cyclic process of analyze, improve and simulate

LangChain-SynData-RAG-Eval★ 40

LangChain, Llama2-Chat, and zero- and few-shot prompting are used to generate synthetic datasets for IR and RAG system evaluation

EvalsforAgentsInterop★ 40

Redefine agentic evaluation in enterprise AI. By simulating realistic scenarios, these evals enables rigorous, multi-dimensional assessment of LLM-powered productivity agents.

ampere★ 40

Observable AI cognition in Kotlin Multiplatform. Every agent decision emits a structured event.

evalmonkey★ 40

CLI for agent builders to benchmark & chaos test your AI Agents. Text, Voice, Code supported.

tokenmaxxing★ 40

Social leaderboard for LLM token usage — sync your agent usage with the CLI, climb the leaderboard

voicetest★ 40

Test harness for voice agents. Import from Retell, VAPI, Bland, LiveKit. Run autonomous simulations. Evaluate with LLM judges.

proofagent-harness★ 40

Open-source test harness for AI agents. Stress-test production agents with adversarial multi-turn scenarios in CI

DshCockpit★ 40

Desktop cockpit for DeepSeek Harness (dsh): token usage & cost tracking, budget alerts, runtime auto-update with rollback, Quick Ask hotkey, scheduled tasks, session search. Win+macOS. DeepSeek Harness 桌面驾驶舱:成本/用量监控 · 自动更新 · 定时任务

openclaw-dashboard★ 39

Mission Control dashboard for monitoring and managing AI agent fleets. Track costs, performance, context health, security posture, and market intelligence.

datacontext★ 39

Open-source runtime attribution and context observability for data access by AI agents and applications

tokenuse★ 39

A Rust TUI and Desktop app for exploring AI coding tool token and cost usage

Ko-AgentBench★ 39

gradio leaderboard region:us

AgentCheck★ 39

Behavioral testing for AI agents.

dokimi★ 39

Dokimi (Greek δοκιμή, "test, proof") measures whether a skill, MCP doc source, agent profile or repository guidance actually helps a coding agent: paired on/off arm comparisons, novel-vs-known content analysis, and repository scans with graded agent runs.

pi-insights★ 38

Personal usage analytics for Pi coding agent. Temporal-aware insights, context-aware suggestions, model efficiency analysis.

C3-Benchmark★ 38

C^3-Bench: The Things Real Disturbing LLM based Agent in Multi-Tasking

MASArena★ 38

A comprehensive framework for benchmarking single and multi-agent systems across a wide range of tasks—evaluating performance, accuracy, and efficiency with built-in visualization and tool integration.

Openclaw-Jarvis-dashboard★ 38

OpenClaw 贾维斯看板是一个面向 OpenClaw 多 Agent 系统的实时作战指挥中心。 它提供系统健康、Agent 运行态、任务管理与多模型用量看板(MiniMax / OpenAI / Gemini / GLM)的统一视图,支持本地 API、实时刷新与可视化配置,帮助你在一页内快速完成监控、诊断和操作。

llm-blender★ 38

LLM-Blender: Ensembling framework that maximizes LLM performance via pairwise ranking. Employs PairRanker to rank candidates and GenFuser to merge outputs, generating superior responses by combining the diverse strengths of multiple open-source models.

ccclub★ 38

Know how much Claude Code your friends are burning through. Create a group, share a code, compare usage.

Benchmark-Agent★ 38

Benchmark Everything Everywhere All at Once, a fully autonomous agentic system for benchmark construction and customization.

PAST-Bench★ 38

PAST-Bench: Benchmarking the Foundations of Recursive Self-Improvement in Personal Agents

money-agent★ 38

Overnight experiment: can an AI agent make money, and can you trust what it reports? The agent produces claims; a separate verifier it cannot invoke produces the facts.

explicit-edit-benchmark★ 38

Benchmark coding agents and agent harnesses on 226 deterministic file-editing tasks with byte-exact verification and a public leaderboard.

agentcap★ 38

A framework to capture, analyse and export agentic sessions

PTXBench★ 38

PTXBench: an open-source environment to evaluate GPU kernel languages for LLM agents

openalerts★ 37

An alerting layer for agentic frameworks

LongTracer★ 37

Detect hallucinations in LLM responses. Verify every claim against source documents using hybrid STS + NLI. Works with LangChain, LlamaIndex, or any RAG pipeline. pip install longtracer

hermescheck★ 37

Hermes Agent-focused architecture and runtime health checks

laravel-ai-orbit★ 37

Standalone observability dashboard and developer playground for the Laravel AI SDK. Zero build steps. Telescope for your AI agents.

More in Agent frameworks & infrastructure