Nonlinear Attention Enables Inverse-Depth Scaling

Nonlinear attention mechanisms allow models to focus selectively on relevant tokens, enabling parallel learning of strong and weak spectral directions. This mechanism explains why loss scales with inverse depth, overcoming the limitations of linear-attention models that rely on global spectral strength.