Psychometric Audit Reveals MMLU Measures Retrieval Rather Than Reasoning

A psychometric analysis of 1,000 open-weights models shows MMLU scores primarily reflect factual retrieval capacity rather than reasoning. The study demonstrates that MMLU conflates separable constructs and that difficulty is driven by text-extractable structural complexity.