Agent frameworks & infrastructure
Observability & evaluation
5,667 agents · most starred first
Observability & evaluation, page 25 of 57
Get a bigger model to check your smaller model daily runs and improve it on a daily basis
gradio evaluation agents llm observability region:us
gradio region:us
gradio region:us
gradio region:us
gradio region:us
gradio region:us
streamlit region:us
docker leaderboard region:us
streamlit region:us
gradio region:us
gradio region:us
gradio region:us
gradio region:us
gradio leaderboard region:us
gradio region:us
docker streamlit region:us
gradio region:us
gradio 日本語 Japanese leaderboard language:日本語 language:Japanese region:us
docker streamlit region:us
static region:us
gradio region:us
static llm-ranking leaderboard llm-leaderboard evaluation human-preference humor arena
gradio region:us
gradio region:us
streamlit region:us
gradio mcp-server-track mcp-server region:us
streamlit region:us
gradio region:us
Offline plan drift scoring for LLM agent traces
AI agents built spaCy model recommenders, then a fresh AI peer-review panel rejected all three
CortexOps is the CI reliability layer for AI agents — golden datasets, tool-call gates, and regression diffs that block bad deploys before users see them.
Open-source, privacy-first observability runtime for multi-agent AI systems across frameworks, providers, OpenTelemetry, and durable transports.
Real-world evaluation framework for AI coding agents — measures safety, containment, cost, and autonomy beyond just correctness.
Four-value evaluation for agentic AI loops. Replace pass/fail with NO/YES/MAYBE/IFF.
Decision layer samples for diagnostic reasoning and RoboCasa scenes for LLM agent evaluation
Offline, single-file rubric for validating AI outputs — routes the right checks, writes the prompt, catches the stale number
Evaluation framework for LLM-agent reliability — scores intermediate reasoning steps, not just final outputs. Demonstrated on a resume–JD screening agent with a human-in-the-loop gate. Trajectory-level scorers, pass^k stability, and prompt-injection rounds across 2 model providers.
Reference implementation and pattern library for proof gates: agent-authored verification contracts resistant to sycophancy and distribution skew. Reproducibility artifact for the paper 'Proof Gates' (Whittaker 2026).
Tools for running AI agents in production — safe, observable, cheap to operate.
static region:us
static leaderboard region:us
Hands-on, reproducible benchmarks of AI agent tooling — same machine, same tasks, full scripts
Pre-registered benchmark: HTML dialect + whole-tree rewrite (barkup) vs JSON, granular mutation tools, and two patch dialects, for LLM agents editing typed trees. 9,600 scored runs across 4 models. Findings in REPORT.md.
Explain failed DeepSeek Harness agent runs locally, then produce a redacted recovery plan. No model required.
Deterministic CI regression testing for policy drift across model-visible tools, semantic validation, SQL compilation, database containment, and result release.
[DCOSS-IoT 2026 22nd] Stable Behavior, Limited Variation: Persona Validity in LLM Agents for Urban Sentiment Perception
Nook is a progression system for AI agents. Verified work generates immutable sparks that evolve your agents over time
Noetrium: Reproducible Research Infrastructure for AI Agents - experiments, models, environments, evidence, recovery, evaluation, and verification.
empirical research framework designed to analyze multi-agent Large Language Model (LLM) conversations as observable, continuous-state dynamical systems.
DSV4 behavior-discrimination benchmark: 81 lightweight zero-dependency pytest grading tests (d10/d11/d12/t2/t3/t4/v4) for datapipe — discriminates harness/agent/persona effects that public tests saturate on
No Promises Without Proof. A dependency-free discipline that forces agents to pair every success claim with evidence and a proof state (OBSERVED/INFERRED/UNVERIFIED/BLOCKED), then derives the completion verdict deterministically.
docker streamlit plotly llmops observability telemetry evaluation model-monitoring
A pi agentic extension that stops weak local LLMs from confidently hallucinating. Auto-dispatches a web-browsing librarian sub-agent to verify API defaults, flags, versions & docs from live sources, returning cited evidence. Works with llama.cpp / Ollama. No cloud required.
AtFlows is a local observability tool for LLM applications. Point your SDK at it, see your costs, tokens, and latency in real-time.
A benchmark measuring AI agent reliability on paired Amharic and English tasks in simulated Ethiopian workflows.
How modern LLMs answer russian property management questions?
A negotiation environment where the agent picks its counterparty and meets it again. Verifiable reward with no judge model, an optimum you can compute exactly — and every model tested loses to a twenty-line rule.
Measure your agent harness, find where it wastes the model, and prove the fix worked. Harness-agnostic, agent-agnostic, zero dependencies. Reference implementation of HTP-1.
LLM agent harness with versioned prompts, runtime guardrails, automated evals and a CI quality gate. Runs offline with no API key.
Adaptive Optimization Framework for AI Agents using Reinforcement Learning. Automatically optimize agent behaviors, reduce costs by 30%, and improve performance through Q-learning and data-driven prompt engineering.
A working Demo that acts as classifier to classify linkedin post which inside uses jev by Typesafe.ai
一个跑得起来的端侧大模型能力验证框架:它不训练模型——参数永远不动—— 而是让 agent 在真实数据库环境里自己出题、自己取数、自己被判卷、 把跑通的做法结晶成技能,再用消费线检验:换一个模型接进来,已学技能还在不在、 答案还判不判得对——差距就是该模型在这个环境里的真实能力。 习得是手段,验证是目的。 被测对象是接入的 LLM;技能库、判官、 防幻觉闸都是量具。这套设计源自 Voyager(Wang et al., 2023)开创的范式: 把"学习"从改模型权重换成改可执行的技能库——环境反馈不回传梯度, 而是回传一个判卷结论;经验不是存成样本对,而是结晶成 可检索、可组合、可重放的程序。 原版在 Minecraft 里验证,本仓库把它搬进真实数据库世界并做成了 按场景隔离、可插拔泛化
Integrity scenarios for agent runtimes, distilled from real fleet incidents — deterministic fault-injection checks against real SDKs
Open RDelta/Babel Index framework for LLM agents and world-systems: detect context degeneration, agent loops, tool fragility, and collapse with NEOTH.
Evaluation framework for A/B test analysis agent.
QED Proof Python SDK (pip install qed-proof): submit an AI agent's claimed work, get an independently verified verdict, and verify the signed receipt offline.
Open Benchmarks and reproducible evaluation harnesses for Shadow AI Agent Discovery & Inventory
LLM personality evaluation and monitoring platform
Find where coding models fail across providers and coding tools.
Identify which LLM produced given answers — based on single-token behavioral fingerprints (Bruckner 2026, arXiv:2607.10252). 170 models, 10 tasks, 4 languages.
Coding-agent trajectory portability: Claude Code to ATIF v1.7 with fidelity accounting, redaction, and historical observability projection.
STRATA-Bench: Spatial-Temporal Retrieval, Alignment, Transparency & Audit — a benchmark for AI agents on fragmented geographic and longitudinal market intelligence.
A live, contamination-resistant benchmark for evaluating LLM agents on U.S. macroeconomic nowcasting.
Git-native regression testing for LLM agent tool loops. Assert on the trajectory — the ordered tool calls — not the final prose. Local-first, deterministic, no API key in CI; exit codes are the API.
Benchmark the same task across agent frameworks (AutoGen, LangGraph) and LLMs — quality, latency, and cost in one table.
CI for tool-using agents: deterministic eval suites with record/replay, contracts, budgets, and regression merge gates.
An eagle's nest, literally the high vantage point it watches everything from
Hermes Agent ops: 35 modules for running a self-hosted LLM agent 24/7 — cron reliability, cost guards, context compaction, probes, skill scanners.
Regression tests for LLM agents: replay a frozen conversation through today's code, test only the next user line. Stale detection, code-judged checks, guard attribution, pass^k and significance tests, pytest plugin.
An open benchmark and reference for evaluating AI agent memory systems.
A self-contained dashboard showing how many tokens (and dollars) claude-mem and Graphify save you. Auto-detects both tools, opens from disk, honest by default.
Local-first lab for testing AI agent workflows end to end: emulated Slack (signed events, buttons), local S3/SQS, deterministic authorization, and independent evidence instead of self-grading. Optional Google AX (Agent Executor) + Agent Substrate lane on GKE/gVisor, reported per assertion. Independent; not affiliated with Google.
Game-style HUD overlay for AI agent observability. Live cards per agent, drop-in for LangChain / OpenAI Agents / Claude Agent SDK / CrewAI / AutoGen. Skin marketplace for creators.
Open Benchmarks and reproducible evaluation harnesses for Shadow AI Agent Discovery & Inventory
gradio region:us
gradio region:us
streamlit region:us
streamlit region:us
gradio region:us
gradio region:us
gradio region:us
gradio region:us
gradio region:us
streamlit region:us
gradio region:us
gradio region:us
gradio leaderboard region:us
gradio region:us
static region:us