TelemetrySuffBench reveals gaps between failure detection and origin diagnosis
August 11, 2026
TelemetrySuffBench evaluates whether execution traces provide enough information to locate failure origins in multi-component agent systems. While frontier models achieve up to 97.2% origin-step accuracy with full telemetry, standard OpenTelemetry-compatible views show nearly 100% detection F1 but drop to 0.5% accuracy for fault localization.
HOW THIS AFFECTS YOU
●
builderYou may need more granular traces than standard OpenTelemetry views to debug agent failures.
●
researcherThis establishes a benchmark for separating detection from localization in agentic workflows.