Measuring Task Depth in Long-Horizon Agent Benchmarks
October 31, 2026
This study argues that task difficulty in long-horizon benchmarks is better measured by 'depth'—the required sequence of operations—than by turn count. The researchers found that turn counts often reflect model habits rather than true task complexity.
HOW THIS AFFECTS YOU
●
builderAvoid designing benchmarks that reward models for having specific turn-count habits.
●
researcherAdopt depth-based metrics to more accurately evaluate agent reasoning capabilities.