Audit of Agent Leaderboards Reveals Grading and Instrument Errors
November 4, 2026
An audit of the HAL and tau2-bench leaderboards identifies critical flaws, including incomplete grading of conversations and double-counting model calls in cost metrics. Re-grading archived trajectories with corrected evaluators significantly reorders agent performance rankings.
HOW THIS AFFECTS YOU
●
builderEnsure your agent evaluation pipeline accounts for grading-triggering conversation endings to avoid artificial performance gaps.
●
researcherBe skeptical of current agent benchmarks until grading transparency and completeness are verified.