Chess Explanations Fail to Accurately Reflect LLM Decision Logic
September 22, 2026
Testing on 200 Lichess endgame puzzles reveals that LLM-generated chess move explanations do not reliably reflect the underlying decision process. Under strict masking, explanations provided minimal gains over board state alone, suggesting reasoning text is often unfaithful to the model's actual token-level probabilities.
HOW THIS AFFECTS YOU
●
builderYou should be cautious about using generated explanations as a proxy for understanding model logic in production.
●
researcherThis highlights the critical gap between linguistic fluency and actual reasoning faithfulness in LLMs.