Agent frameworks & infrastructure
Observability & evaluation
5,667 agents ยท most starred first
Observability & evaluation, page 22 of 57
Final project for the Hugging Face AI Agents course testing smolagents on the GAIA benchmark
CAST observability UI โ React 19 + Express dashboard for cast.db session analytics, agent runs, and hooks
Okahu demo chatboth with Langchain and OpenAI
Tutorial showing how to do RAG evals with Parea AI and LangChain
Track, visualize, and optimize LLM API spending. Monitor OpenAI & Anthropic costs per feature, detect waste, suggest savings. Zero-config Python profiler.
๐ Production-grade, full-stack agentic ecosystem engineered with DB-first approach using Turso & Drizzle, Generative UI built with Next.js & Server-Sent Events (SSE), a robust LLM-as-Judge evaluation suite. Fully containerized and orchestrated via Kubernetes & Helm, leverages modern DevOps with GitHub Actions & GHCR.io scalable cloud-native deploy
Agent and LLM API performance monitoring via network packet probe. Measures performance of OpenClaw, Claude, Codex, DeepAgents and more โ deployed on the provider side, no SDK changes required.
Developer CLI for AgenticGoKit - scaffolding, tracing, and debugging AI agent workflows
Develop, Test, and Benchmark Agentic Workflows before they hit production.
KodaAI is a performance analytics platform that quantifies the efficiency gains provided by AI-assisted development using Kiro. It bridges the gap between human estimation (Jira) and machine-accelerated output (GitHub/Kiro) to calculate a "Speed Multiplier" for software teams.
๐คซ Simulate LLM API token consumption to hit your KPI targets. Supports 25+ platforms (OpenAI, Claude, Kimi, DeepSeek...) with work-simulation mode.
A simple Postgres logger for OpenAI endpoint calls
Quick protocol checks for AI and LLM API endpoints.
A benchmark study analyzing cost and token efficiency across 14 LLMs from 5 providers โ comparing price-per-token, latency, and accuracy to surface the most cost-effective models for real-world use.
OpenAI Prometheus Exporter
See How Big Exactly A 128k Token Text Is
Turn your MacBook notch into a live Claude AI usage monitor โ Dynamic Island-style, native Swift, zero telemetry.
Real-time spending alerts for Claude Code โ budget limits, loop detection, context warnings, and plan cost estimation. Works in CLI, VS Code, Cursor, and JetBrains.
Traceprompt is an open-source SDK that seals every LLM call and exports write-once, read-many (WORM) logs auditors trust.
Privacy-first Claude Code burn-rate analyzer. Predict weekly limit hits, spot model-swap savings.
Published PyPI package for ArXiv embedding benchmarks, retrieval evaluation, and scientific RAG experiments.
A comprehensive evaluation toolkit for assessing Retrieval-Augmented Generation (RAG) outputs using linguistic, semantic, and fairness metrics
RAG Evaluation using RAGAS
Find exactly which parts of your LLM output are hallucinated. Claim-level factuality verification with NLI, calibrated confidence, and pipeline diagnostics for RAG systems.
A library for evaluating Retrieval-Augmented Generation (RAG) systems
MEMERAG: A Multilingual End-to-End Meta-Evaluation Benchmark for Retrieval Augmented Generation
End-to-end GenAI framework with built-in evaluation, security, and modular productivity tooling.
ATLAS: A Measurable Reliability Framework for AI Systems
This project(RAG) focuses on operationalizing LLMs by integrating OpenAI, MLflow, FastAPI, and RAGAS for evaluation. It allows users to deploy and manage LLMs, track model runs, and log evaluation metrics in MLflow. The project also features MLflow traces that logs all the user inputs ,responses ,retrieved contexts ,and other essential metrices.
TypeScript SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)
๐ง Discover and evaluate advanced benchmark datasets for Large Language Model agents to enhance performance assessment in real-world tasks.
Playwright for AI Agents. Test what your agent DOES, not what it SAYS. YAML-first behavioral testing. Catch PII leaks, tool abuse, step explosions. 3200+ tests.
Self-hosted AI agent observability with tool-call tracing and decision tree visualization
Local-first causal lineage and observability for AI-assisted development
htop for AI coding agents
A travel agent python app built with CrewAI framework using GPT model from OpenAI to be run locally or on CI/CD with Github Actions.
Rag Badger is a minimal open source toolkit to evaluate RAG based systems efficiently. Best suited for evaluating RAG pipelines with the help of Ollama's local LLMs and embedding models.
Multi-provider LLM pipeline + open benchmark measuring when each frontier model runs out of distinct ideas. 10 LLMs, 16ร cost-per-unique-idea spread for the same shipped quality.
VS Code extension that shows your Claude Code rate limit usage in the status bar. Tracks 5-hour session and weekly limits with live progress bars, color alerts, and reset countdowns via passive diagnostics_channel interception โ zero extra API calls.
Detect behavioural drift between LLM versions before you upgrade. Compare model responses, classify regressions, and generate migration reports with validated prompt patches.
See what your AI is thinking. See what it costs. One binary, zero config.
Self-hosted multi-device Claude Code / Codex usage aggregation server โ PromQL, Grafana, JWT
Python runtime checks for AI agents: budgets, loops, retries, and local traces. Zero runtime dependencies. MIT.
Side-by-side A/B testing for CLAUDE.md, skills, and agents, with live transcripts and AI-judged results.
A modern, real-time quota monitor for MiniMax API. Features a semi-transparent floating UI, precise RPM tracking, smooth animations, and persistent configuration.
BenchClaw adapters for LangChain, LlamaIndex, CrewAI, Ollama Open WebUI, LobeChat, n8n, Dify, Continue.dev and more. Lets any agent framework submit to the P2PCLAW leaderboard.
[ICML'26] AgentWebBench: Benchmarking Multi-Agent Coordination in Agentic Web
Skills that guide AI coding agents to integrate with the Adaline platform โ send traces, manage prompts, run evaluations, fetch deployments, and more. Compatible with Cursor, Claude Code, Codex, Windsurf, and 40+ other agents.
Prediction-market agent arena for AI agent evaluation, paper trading, practice rounds, contests, and leaderboard-based battle testing.
Proof Gradient is the agent evolution protocol where every run leaves proof, every proof selects intelligence, and every selected artifact evolves the network.
Deterministic $0 pytest assertions for AI agents โ catch the agent that quietly stopped calling its tools, and cost blowups, offline in CI.
Single-binary, MLflow-API-compatible experiment tracker with first-class LLM trace support. 143ร faster cold start than MLflow.
๐ฉบ RAG Doctor โ Open-source diagnostic tool for Retrieval-Augmented Generation (RAG) systems. Analyzes codebases to detect architectural issues in LLM pipelines such as missing retrieval, bad chunking, embedding mismatches, and vector database misuse.
Make sure AI agents actually do what they claim, no more fake successes.
Local-first Harness Operating System for defining, evaluating, evolving, versioning, and deploying agent harnesses.
Benchmark methodology, task sets, and evaluation results for RAยฒR
A/B evaluate any LLM task with and without Ejentum cognitive injection. n8n workflow + TypeScript module.
Multi-agent synthetic-audience evaluation โ parallel eval agents that score content and code before ship
Automated chatbot evaluation for generated natural language responses. Published in NAACL 2019 and powers chateval.org.
I built AgentWatch to solve a problem I had: AI agents burning tokens and money behind my back. Now I can monitor everything in real-time. Feedback welcome! ๐ค๐ธ
Find out where your Claude Code tokens actually go โ exact spend + recoverable waste, measured from your local session logs. Zero deps, zero network.
Local-first analytics dashboard for Claude Code: cost, tokens, models, tools, heatmap, leaderboards & Claude Wrapped โ one HTML file, zero deps, no telemetry.
See what your AI-coding session actually cost โ and what is eating your context. Local, read-only CLI for Claude Code.
Audit your Claude Code token and cost spend and monitor what your agents are doing.
Cost control for teams building on AI. Enforce spend budgets before the call goes out. Self-hostable dashboard (Next.js).
macOS menu-bar & Windows system-tray app showing real-time usage limits for Claude, Codex & GitHub Copilot.
See where LLM money goes, why, and how to cut waste. Local-first FinOps + spend intelligence for Claude Code & the Anthropic API. Pure stdlib, zero deps.
Migrate LLM apps between models safely โ then benchmark old vs. new to prove the upgrade. Agent skill for behavior-preserving migrations + a runnable eval harness (three-arm leaderboard, prompt sweeps). Works in Claude Code, Codex & Cursor.
Stop arguing about prompts. Measure them. A reproducible, statistically-honest benchmark harness for Claude Code workflows (CLAUDE.md, prompts, memory). Wilson CIs, significance-tested verdicts, test-based grading, record/replay. Zero deps.
NeurIPS 2025 D&B | RAG-IGBench: benchmark for RAG-based interleaved image-text generation.
Reproducible benchmarks for execution-intent memory in long-horizon AI coding agents. ID-RAG cross-corpus matrix + LongMemEval-S subset; BYO API keys.
A lab for studying how LLMs behave when offered a set of fake tools
[IJCAI'26] Visualizing Deep Agents in Long-Horizon Tasks: Towards Explainable and Trustworthy Agentic AI
Dead-simple, local-first, framework-agnostic agent observability.
Hermes Monitor โ Pixel-art real-time AI agent monitoring dashboard for multi-persona teams
docker region:us
docker openenv rag reinforcement-learning multi-agent aerospace legal-research rl-environment
streamlit region:us
gradio region:us
gradio region:us
gradio region:us
gradio region:us
gradio region:us
static region:us
gradio region:us
streamlit region:us
gradio region:us
Circuit breaker + hierarchical dollar budget for LangGraph agents. In-process, no proxy.
Seamless LLM Lineage for DataHub with LangChain and LangSmith
SAGE (Session Awareness & Guidance Engine) - a passive, read-only judge for fleets of parallel AI coding sessions. It watches and answers; it never edits, spawns, or blocks.
OpenTelemetry + Wireshark + Cloudflare for AI Agents. Monitor, inspect, secure, replay, and optimize all traffic between Users, AI Agents, LLMs, and MCP Servers.
Native macOS menubar app that tracks Claude Code usage costs, quotas, and session insights โ processed locally on your Mac.
Share & pool Claude usage across your team โ a free macOS menu-bar app with live quota, burn-rate alerts, and end-to-end-encrypted borrowing.
macOS menu bar app for Claude Code token usage visualization
CLI for peak AI inference performance. Analyze code for latency, throughput, reliability issues. BYOK mode.
Analyze Claude Code token usage and costs from local conversation logs and statusline data
Desk display showing Claude Code & Codex usage limits (5h/7d) on a Waveshare ESP32-S3 round AMOLED. LVGL firmware + Python host bridge.
Observability layer for ML/RAG pipelines. Traces every stage, records what was retrieved, what got dropped, and why. Python SDK + FastAPI + React dashboard.
Evaluation rubric, fixtures, and scoring tools for JakMa's grounded-retrieval pipeline.