LM Head Gradient Bottleneck Suppresses 95-99% of Gradient Norm

The output layer in language models creates an optimization bottleneck where backpropagating V-dimensional gradients through a rank-D linear layer compresses signal. Empirical tests show the bottleneck suppresses 95-99% of the gradient norm, making trivial patterns unlearnable and altering update directions for the majority of parameters.