Gradient Starvation prevents Transformers from learning deep semantic dependencies

A new mechanistic framework identifies 'Gradient Starvation,' where early optimization suppresses error signals for sparse semantic dependencies in favor of surface statistics. This phenomenon explains the sudden phase transition in structural reasoning and provides a theoretical basis for why Chain-of-Thought (CoT) improves reasoning by externalizing intermediate steps.