Attention Mass Predicts Retrieval Failure Over Context Length
November 4, 2026
Analysis of five open models (0.5B to 3B parameters) shows that retrieval failure is driven by competitive information rather than context length. Attention mass left on evidence explains up to 58% of loss variance, while log length explains less than 8%.
HOW THIS AFFECTS YOU
●
builderYou should focus on reducing information competition in your RAG pipelines rather than just increasing context windows.
●
researcherThis challenges the assumption that context length is the primary bottleneck in long-context retrieval.