Diagnosing Benchmark Failure via Empirical Difficulty Measures

A new diagnostic framework identifies measure mismatch and coverage gaps in LLM evaluations by treating benchmarks as empirical measures over difficulty. Analysis of 25 benchmarks across 7 domains shows many leaderboards overweight specific difficulty regions, potentially masking training progress.