Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 25 of 57

glean★ 2

Get a bigger model to check your smaller model daily runs and improve it on a daily basis

AI-agent-Evaluation-pipeline★ 2

gradio evaluation agents llm observability region:us

Judges-copilot★ 2

gradio region:us

testing_llm★ 2

gradio region:us

llm-vs-llm★ 2

gradio region:us

LLMRiddles★ 2

gradio region:us

LLMRiddlesChatGLMEN★ 2

gradio region:us

Lambda-LLM-Calculator★ 2

streamlit region:us

Yet_Another_LLM_Leaderboard★ 2

docker leaderboard region:us

alpaca-eval-explorer★ 2

streamlit region:us

leaderboard_llm_price★ 2

gradio region:us

llm_leaderboards360★ 2

gradio region:us

Two-phase_Fine-tuned_LLMs-CreativityAutoEvaluation★ 2

gradio region:us

llm-ppls★ 2

gradio region:us

llm-hkmmlu-leaderboard★ 2

gradio leaderboard region:us

llm-evaluation-framework★ 2

gradio region:us

LMVal-Multi-Metric-LLM-Evaluation★ 2

docker streamlit region:us

LLM-Propensity-Evals★ 2

gradio region:us

open-japanese-llm-leaderboard★ 2

gradio 日本語 Japanese leaderboard language:日本語 language:Japanese region:us

LLM-Compute-Optimal-Estimator★ 2

docker streamlit region:us

open-telecom-llm-leaderboard★ 2

static region:us

llmlagbench★ 2

gradio region:us

llm-humor-ranking-leaderboard★ 2

static llm-ranking leaderboard llm-leaderboard evaluation human-preference humor arena

Persian-linguistic-llm-leaderboard★ 2

gradio region:us

Mushroom_Hunting_in_Arabic_LLMs★ 2

gradio region:us

RAG-App-Evaluator★ 2

streamlit region:us

RAGEval-Studio★ 2

gradio mcp-server-track mcp-server region:us

compare-gpt-models★ 2

streamlit region:us

Leaderboard_dev★ 2

gradio region:us

agent-plan-drift-meter★ 2

Offline plan drift scoring for LLM agent traces

inference_conference★ 2

AI agents built spaCy model recommenders, then a fresh AI peer-review panel rejected all three

cortexops★ 2

CortexOps is the CI reliability layer for AI agents — golden datasets, tool-call gates, and regression diffs that block bad deploys before users see them.

causentra★ 2

Open-source, privacy-first observability runtime for multi-agent AI systems across frameworks, providers, OpenTelemetry, and durable transports.

agentbench-rw★ 2

Real-world evaluation framework for AI coding agents — measures safety, containment, cost, and autonomy beyond just correctness.

verdict4★ 2

Four-value evaluation for agentic AI loops. Replace pass/fail with NO/YES/MAYBE/IFF.

rlenv★ 2

Decision layer samples for diagnostic reasoning and RoboCasa scenes for LLM agent evaluation

diligence-agent★ 2

Offline, single-file rubric for validating AI outputs — routes the right checks, writes the prompt, catches the stale number

trajectory-eval-harness★ 2

Evaluation framework for LLM-agent reliability — scores intermediate reasoning steps, not just final outputs. Demonstrated on a resume–JD screening agent with a human-in-the-loop gate. Trajectory-level scorers, pass^k stability, and prompt-injection rounds across 2 model providers.

proof-gate-patterns★ 2

Reference implementation and pattern library for proof gates: agent-authored verification contracts resistant to sycophancy and distribution skew. Reproducibility artifact for the paper 'Proof Gates' (Whittaker 2026).

shua-labs★ 2

Tools for running AI agents in production — safe, observable, cheap to operate.

sally-llm-leaderboard★ 2

static region:us

llm_leaderboard_fr★ 2

static leaderboard region:us

agent-tool-benchmark★ 2

Hands-on, reproducible benchmarks of AI agent tooling — same machine, same tasks, full scripts

barkup-bench★ 2

Pre-registered benchmark: HTML dialect + whole-tree rewrite (barkup) vs JSON, granular mutation tools, and two patch dialects, for LLM agents editing typed trees. 9,600 scored runs across 4 models. Findings in REPORT.md.

dsh-postmortem★ 2

Explain failed DeepSeek Harness agent runs locally, then produce a redacted recovery plan. No model required.

policystrata★ 2

Deterministic CI regression testing for policy drift across model-visible tools, semantic validation, SQL compilation, database containment, and result release.

MLLMs-persona-evaluation★ 2

[DCOSS-IoT 2026 22nd] Stable Behavior, Limited Variation: Persona Validity in LLM Agents for Urban Sentiment Perception

nook★ 2

Nook is a progression system for AI agents. Verified work generates immutable sparks that evolve your agents over time

Noetrium★ 2

Noetrium: Reproducible Research Infrastructure for AI Agents - experiments, models, environments, evidence, recovery, evaluation, and verification.

NOMAD★ 2

empirical research framework designed to analyze multi-agent Large Language Model (LLM) conversations as observable, continuous-state dynamical systems.

dsv4-behavior-bench★ 2

DSV4 behavior-discrimination benchmark: 81 lightweight zero-dependency pytest grading tests (d10/d11/d12/t2/t3/t4/v4) for datapipe — discriminates harness/agent/persona effects that public tests saturate on

Cognitive-Friction-Toolkit★ 2

No Promises Without Proof. A dependency-free discipline that forces agents to pair every success claim with evidence and a proof state (OBSERVED/INFERRED/UNVERIFIED/BLOCKED), then derives the completion verdict deterministically.

llmops-telemetry-command-center★ 2

docker streamlit plotly llmops observability telemetry evaluation model-monitoring

pi-ultra-scout★ 2

A pi agentic extension that stops weak local LLMs from confidently hallucinating. Auto-dispatches a web-browsing librarian sub-agent to verify API defaults, flags, versions & docs from live sources, returning cited evidence. Works with llama.cpp / Ollama. No cloud required.

atflows★ 2

AtFlows is a local observability tool for LLM applications. Point your SDK at it, see your costs, tokens, and latency in real-time.

FIDEL-Agents★ 2

A benchmark measuring AI agent reliability on paired Amharic and English tasks in simulated Ethiopian workflows.

ru-prop-management-eval★ 2

How modern LLMs answer russian property management questions?

nego-eval★ 2

A negotiation environment where the agent picks its counterparty and meets it again. Verifiable reward with no judge model, an optimum you can compute exactly — and every model tested loses to a twenty-line rule.

harness-tuner★ 2

Measure your agent harness, find where it wastes the model, and prove the fix worked. Harness-agnostic, agent-agnostic, zero dependencies. Reference implementation of HTP-1.

agent-quality-kit★ 2

LLM agent harness with versioned prompts, runtime guardrails, automated evals and a CI quality gate. Runs offline with no API key.

adaptiq★ 2

Adaptive Optimization Framework for AI Agents using Reinforcement Learning. Automatically optimize agent behaviors, reduce costs by 30%, and improve performance through Q-learning and data-driven prompt engineering.

Postmark★ 2

A working Demo that acts as classifier to classify linkedin post which inside uses jev by Typesafe.ai

voyager-vlml-mve★ 2

一个跑得起来的端侧大模型能力验证框架:它不训练模型——参数永远不动—— 而是让 agent 在真实数据库环境里自己出题、自己取数、自己被判卷、 把跑通的做法结晶成技能,再用消费线检验:换一个模型接进来,已学技能还在不在、 答案还判不判得对——差距就是该模型在这个环境里的真实能力。 习得是手段,验证是目的。 被测对象是接入的 LLM;技能库、判官、 防幻觉闸都是量具。这套设计源自 Voyager(Wang et al., 2023)开创的范式: 把"学习"从改模型权重换成改可执行的技能库——环境反馈不回传梯度, 而是回传一个判卷结论;经验不是存成样本对,而是结晶成 可检索、可组合、可重放的程序。 原版在 Minecraft 里验证,本仓库把它搬进真实数据库世界并做成了 按场景隔离、可插拔泛化

agent-runtime-integrity-bench★ 2

Integrity scenarios for agent runtimes, distilled from real fleet incidents — deterministic fault-injection checks against real SDKs

delta-kosmologie★ 2

Open RDelta/Babel Index framework for LLM agents and world-systems: detect context degeneration, agent loops, tool fragility, and collapse with NEOTH.

experiment-agent-evals★ 2

Evaluation framework for A/B test analysis agent.

qed-proof-python★ 2

QED Proof Python SDK (pip install qed-proof): submit an AI agent's claimed work, get an independently verified verdict, and verify the signed receipt offline.

Nexus-ShadowBench★ 2

Open Benchmarks and reproducible evaluation harnesses for Shadow AI Agent Discovery & Inventory

lindr★ 2

LLM personality evaluation and monitoring platform

modelstackcheck★ 2

Find where coding models fail across providers and coding tools.

llm-fingerprint★ 2

Identify which LLM produced given answers — based on single-token behavioral fingerprints (Bruckner 2026, arXiv:2607.10252). 170 models, 10 tasks, 4 languages.

trajectory-fidelity-bridge★ 2

Coding-agent trajectory portability: Claude Code to ATIF v1.7 with fidelity accounting, redaction, and historical observability projection.

strata-bench★ 2

STRATA-Bench: Spatial-Temporal Retrieval, Alignment, Transparency & Audit — a benchmark for AI agents on fragmented geographic and longitudinal market intelligence.

LiveMacroEval★ 2

A live, contamination-resistant benchmark for evaluating LLM agents on U.S. macroeconomic nowcasting.

dryfire★ 2

Git-native regression testing for LLM agent tool loops. Assert on the trajectory — the ordered tool calls — not the final prose. Local-first, deterministic, no API key in CI; exit codes are the API.

SmartWrapperOSS★ 2

Benchmark the same task across agent frameworks (AutoGen, LangGraph) and LLMs — quality, latency, and cost in one table.

Runledger★ 2

CI for tool-using agents: deterministic eval suites with record/replay, contracts, budgets, and regression merge gates.

eyrie★ 2

An eagle's nest, literally the high vantage point it watches everything from

Hermes-Agent-Ops★ 2

Hermes Agent ops: 35 modules for running a self-hosted LLM agent 24/7 — cron reliability, cost guards, context compaction, probes, skill scanners.

goldenline★ 2

Regression tests for LLM agents: replay a frozen conversation through today's code, test only the next user line. Stale detection, code-judged checks, guard attribution, pass^k and significance tests, pytest plugin.

agent-memory-benchmarks★ 2

An open benchmark and reference for evaluating AI agent memory systems.

claude-token-savings-dashboard★ 2

A self-contained dashboard showing how many tokens (and dollars) claude-mem and Graphify save you. Auto-detects both tools, opens from disk, honest by default.

google-ax-agent-proof-lab★ 2

Local-first lab for testing AI agent workflows end to end: emulated Slack (signed events, buttons), local S3/SQS, deterministic authorization, and independent evidence instead of self-grading. Optional Google AX (Agent Executor) + Agent Substrate lane on GKE/gVisor, reported per assertion. Independent; not affiliated with Google.

AgentHUD★ 2

Game-style HUD overlay for AI agent observability. Live cards per agent, drop-in for LangChain / OpenAI Agents / Claude Agent SDK / CrewAI / AutoGen. Skin marketplace for creators.

Nexus-ShadowBench★ 2

Open Benchmarks and reproducible evaluation harnesses for Shadow AI Agent Discovery & Inventory

llms_compare★ 1

gradio region:us

open_llm_leaderboard★ 1

gradio region:us

LLMHallucination_Leaderboard★ 1

streamlit region:us

LLMs_Test_Lab★ 1

streamlit region:us

LLM_leaderboard★ 1

gradio region:us

Onpremise_LLM_Normal_Detection★ 1

gradio region:us

data_only_open_llm_leaderboard★ 1

gradio region:us

LLM_leaderboard_en★ 1

gradio region:us

LLM-Training-Cost-Calculator★ 1

gradio region:us

LLM-Visualization★ 1

streamlit region:us

neubla-llm-evaluation-board★ 1

gradio region:us

maven_2024_llm_eval★ 1

gradio region:us

LLm-benchmark-2★ 1

gradio leaderboard region:us

HF-Serverless-LLM-Inference-API-Status★ 1

gradio region:us

leaderboard★ 1

static region:us

More in Agent frameworks & infrastructure