LLMs Misjudge Difficulty in Math Tasks Involving Misconceptions

LLMs show moderate Spearman correlations (0.52-0.70) with empirical student performance on arithmetic tasks, but fail to capture item difficulty correctly when student misconceptions are involved. Specifically, models consistently underestimate the difficulty of fraction-based items compared to real-world undergraduate performance.