Agent frameworks & infrastructure

Observability & evaluation

5,667 agents · most starred first

Observability & evaluation, page 26 of 57

RAG-Evaluator1★ 1

streamlit region:us

RAG-Evaluation-with-Custom-metrics★ 1

streamlit region:us

AB_Testing_RAG★ 1

docker region:us

Langgraph_RAGAS★ 1

streamlit region:us

RAG_Evaluator★ 1

gradio region:us

claude-token-counter★ 1

static region:us

ciagent★ 1

Find out whether your AI agent's evals are lying to you: stability reports with flip attribution, LLM judge audits, and deterministic checks mined from your KB.

Agentic_Regression_Analysis_8_Agents_1_Pipeline★ 1

gradio region:us

agentwatch★ 1

Multi-agent observability: cascade failure detection, heartbeat monitoring, cross-agent tracing, and forensic replay

agentskeptic

Catch silent failures in agent workflows by verifying real system state

future-agi

Future AGI review: compare features, pricing, use cases, access model, and alternatives for this Observability agent in 2026. Most accurate evaluation agents that work across all modalities

maxim-ai

Maxim AI review: compare features, pricing, use cases, access model, and alternatives for this Observability agent in 2026. Rated 5/5 from 1 verified review. Evaluate and improve AI Agents, faster

confident-ai

Confident AI review: compare features, pricing, use cases, access model, and alternatives for this Observability agent in 2026. The DeepEval LLM Evaluation Platform

arize-ai

Arize AI review: compare features, pricing, use cases, access model, and alternatives for this Observability agent in 2026. AI observability and LLM evaluation platform for monitoring and improving ML models

Athina AI

Athina helps your team build, test, and monitor AI features faster.

Wayfound

Oversee AI agents, ensuring they perform well and stay compliant.

Keywords AI

Monitor and debug your AI applications to ship reliable features faster.

Agenta

Agenta helps teams build, test, and manage reliable AI applications.

Confident AI

Automate AI testing and ensure your AI apps always work well.

Langfuse

Understand, debug, and improve your AI applications easily.

@vercel/detect-agent

Detect if code is running in an AI agent or automated development environment

n8n-nodes-ai-agent-langfuse

n8n community node: AI Agent + Langfuse

@agentbudget/agentbudget

Real-time cost enforcement for AI agent sessions — works with OpenAI and Anthropic

@netmind/arena-cli

Arena CLI - AI Agent Competition Platform

wayfound

Wayfound AI Agent management platform

gunma-agent-dashboard

Admin monitoring and control dashboard for Gunma AI Agent

@mnemom/mnemom

Transparent AI agent tracing

@aiready/agent-grounding

Measures how well an AI agent can navigate a codebase autonomously without human assistance

acp-web-relay

Monitor and control AI agent sessions from any browser.

ai-agent-session-center

A real-time dashboard for monitoring AI agent sessions (Claude Code, Codex) with 3D visualization

@evilmartians/agent-prism-types

TypeScript type definitions and semantic convention constants for [AgentPrism](https://github.com/evilmartians/agent-prism) - AI agent trace visualization library.

agentapprove

AI agent observability and control from your iPhone or Apple Watch

agent-peek

Read-only peek into other AI agent chat sessions.

@ppazosp/agrex

Real-time graph visualizer for AI agent execution flows

agentclash

AI agent evaluation CLI for races, replay, scorecards, and CI regression gates

@korveo/sdk

Local-first AI agent observability SDK (TypeScript)

@hongmaple0820/scale-engine

Executable AI agent governance with workflow gates, evidence, skill/tool orchestration, and traceable HTML artifacts

agentdashpulse

AI Agent Management Dashboard — monitor and manage Claude Code, GitHub Copilot, and other AI agent sessions

@wix/eval-assertions

Assertion framework for AI agent evaluations - supports skill invocation checks, build validation, and LLM-based judging

@evilmartians/agent-prism-data

Data transformation utilities for [AgentPrism](https://github.com/evilmartians/agent-prism) - converting OpenTelemetry and Langfuse traces to UI-ready formats for AI agent trace visualization.

langdrift

Locale-aware eval harness for AI agent behavior

@panctl/desktop

Panopticon Command Deck — AI agent monitoring desktop app

bee-observe-connector

LLM Agent Framework and Observe connector

@loongsuite/opentelemetry-util-genai

OpenTelemetry GenAI utility for standardized telemetry collection across LLM, Agent, Embedding, Tool, Retrieval, Rerank, Memory and more

vibelogger

AI-Native Logging for LLM Agent Development - TypeScript/Node.js Implementation

solid-pipe

A framework for tracing, tracking and evaluating LLM agent pipelines

agentest-kit

⚡️ AI 系统测试框架 — 面向 LLM/Agent 的轻量断言测试工具,AI 界的 Vitest

evalforge

Framework-agnostic LLM agent evaluation harness

@mastra/datadog

Datadog LLM Observability exporter for Mastra - exports tracing data to Datadog's LLM Observability product

vrunai

LLM agent evaluation framework — define agent behaviour in YAML, evaluate against multiple models, compare results

llm-spend-guard

Enforce real-time token budgets and spending limits for OpenAI, Anthropic Claude, and Google Gemini API calls in Node.js

ai-cost-meter

Lightweight, zero-dependency LLM API cost & token usage tracker for OpenAI, Anthropic, Gemini, Mistral, Groq, and DeepSeek

@dan-build/agentstat

Canvas-rendered React component for real-time LLM agent telemetry — live token rates, progress, status transitions, and health scoring.

@pentatonic-ai/ai-agent-sdk

TES SDK — LLM observability and lifecycle tracking via Pentatonic Thing Event System. Track token usage, tool calls, and conversations. Manage things through event-sourced lifecycle stages with AI enrichment and vector search.

e2eval

Middleware-powered eval framework for Claude Code. Intercepts the Anthropic API to observe and scrub tool calls.

agentfuse

Intelligent LLM agent cost optimization runtime. Coming soon.

probitas

Guardrail regression testing for LLM agent tool calls

langsmith

Client library to connect to the LangSmith Observability and Evaluation Platform.

@diabolicallabs/agent-sdk

Cost-tracking middleware for @diabolicallabs/llm-client. Async fire-and-forget ingestion to Agent Spend Dashboard. © Diabolical Labs

@amplitude/ai

Amplitude AI SDK - LLM usage tracking for Amplitude Analytics

@langwatch/server

Run LangWatch locally with one command — `npx @langwatch/server`. Complete LangWatch stack: observability, evaluations, AI gateway, agent simulations.

@goyamegh/agent-health

Agent Evaluation and Observability Framework

agent-inspect

Local evidence debugger and trajectory-test toolkit for TypeScript AI agents — execution trees, TraceContract checks, Evidence v2, and read-only MCP

@posthog/pi

PostHog LLM Analytics extension for pi coding agent

@newrelic/native-metrics

A module for generating metrics from V8.

@fede0089/skill-eval

CLI to evaluate agent skills triggering and functionality

newrelic

New Relic agent

@dwmkerr/mock-llm

Simple OpenAI compatible Mock API server. Useful for deterministic testing of AI applications.

promptfoo

LLM eval & testing toolkit

@agentesting/agentest

Embedded agent simulation & evaluation framework for Node.js/TypeScript

@elastic/apm-rum

Elastic APM JavaScript agent

@empiricalrun/llm

Package to connect and trace LLM calls.

@raindrop-ai/pi-agent

Raindrop observability for Pi Agent — automatic tracing via subscriber or pi-coding-agent extension

@forwardimpact/libeval

Agent evaluation framework — prove whether agent changes improved outcomes with reproducible evidence.

@flink-app/fake-llm-adapter

Fake LLM adapter for Flink AI framework - perfect for demos and testing

@newrelic/browser-agent

New Relic Browser Agent

@reportportal/client-javascript

ReportPortal client for Node.js

@elg/coerce-llm-output

[![CircleCI](https://dl.circleci.com/status-badge/img/gh/taneliang/coerce-llm-output/tree/main.svg?style=svg)](https://dl.circleci.com/status-badge/redirect/gh/taneliang/coerce-llm-output/tree/main) [![codecov](https://codecov.io/github/taneliang/coerce-l

openevals

Much like tests in traditional software, evals are an important part of bringing LLM applications to production. The goal of this package is to help provide a starting point for you to write evals for your LLM applications, from which you can write more c

@inference/tracing

First-party OpenInference-shaped tracing for TypeScript LLM and agent applications on Catalyst by Inference.net.

@e-llm-studio/watch-me-work

Reusable React component for visualizing multi-agent reasoning progress.

@microsoft/m365-copilot-eval

Zero-config Node.js wrapper for M365 Copilot Agent Evaluations CLI (Python-based Azure AI Evaluation SDK)

elastic-apm-node

The official Elastic APM agent for Node.js

@traceloop/node-server-sdk

Traceloop Software Development Kit (SDK) for Node.js

@traceloop/instrumentation-langchain

OpenTelemetry instrumentation for LangchainJS

@langfuse/langchain

Langfuse integration for LangChain

@langfuse/core

Core functions and utilities for Langfuse packages

@langfuse/otel

Langfuse OpenTelemetry export helpers

@langfuse/tracing

Langfuse instrumentation methods based on OpenTelemetry

@langfuse/client

Langfuse API client for universal JavaScript environments

agentevals

[Agentic applications](https://langchain-ai.github.io/langgraph/concepts/agentic_concepts/) give an LLM freedom over control flow in order to solve problems. While this freedom can be extremely powerful, the black box nature of LLMs can make it difficult

langfuse-langchain

LEGACY Langfuse v3 integration for LangChain (maintenance mode). For new projects use the current CallbackHandler from @langfuse/langchain: npm install @langfuse/langchain @langfuse/otel

@braintrust/langchain-js

SDK for integrating Braintrust with LangChain.js

@arizeai/openinference-instrumentation-langchain

OpenInference Instrumentation for LangChain.js

@langfuse/openai

Langfuse integration for OpenAI SDK

langchain-agent-toolkit

Verification, observability, and quality scoring utilities for LangChain tool-calling agents. Includes confidence scoring, data freshness validation, numerical consistency checks, ticker symbol hallucination detection, error categorization, token tracking

@microsoft/agents-a365-observability-extensions-langchain

Microsoft Agent 365 SDK LangChain instrumentation extensions for telemetry and observability

opik-langchain

Opik TypeScript and JavaScript SDK integration with LangChain

@agent-context-kit/langchain

LangChain tools and observability integrations for agent-context-kit

@cloudbase/agent-observability

OpenInference-compatible observability for AG-Kit

More in Agent frameworks & infrastructure