LLM Agent Benchmarks Fail Out-of-Distribution: Predictive Validity Proposed

Aggregate leaderboard scores for LLM agents show rank instability when transferred to out-of-distribution settings, confirmed across 14 parallel implementation studies and 7 prior benchmarks. The paper proposes ranking by predictive validity — correlation between in-sample and out-of-sample performance — rather than aggregate scores. No single existing benchmark covers more than 4–5 deployment-relevant dimensions.