Identifying Critical Logarithmic Scaling to Prevent Attention Rank-Collapse

Analysis of long-context transformers reveals a phase transition in attention where rank-collapse occurs as context length increases. The study provides a theoretical justification for scaling attention scores by a polylogarithmic factor of log n to maintain meaningful token interactions.