Weight Decay Drives the Growth of Massive Activations
October 4, 2026
Analysis of transformer training trajectories reveals that weight decay causally contributes to the emergence and consolidation of massive activations (attention sinks). These sinks concentrate onto a few redundant channels over time.
HOW THIS AFFECTS YOU
●
researcherThis provides deeper insight into the stability and scaling of residual-stream coordinates in transformers.