Evaluating LLM Generalization vs. Memorization via Chess

Using chess as a controlled testbed, this study shows that LLM performance degrades significantly as the density of training-relevant priors decreases. Analysis of GPT, Claude, and Gemini models indicates that many capabilities rely on memorized patterns rather than true reasoning.