Length Is Cheap, Competition Is Not: Attention Predicts Context Rot in Retrieval

Title: Length Is Cheap, Competition Is Not: Attention Predicts Context Rot in Retrieval Source: huggingface Retrieval accuracy drops from ~1.00 to 0.16 when neutral filler is replaced with competitive statements, regardless of position. Attention mass on the target evidence explains up to 58% of loss variance, while log length accounts for only 8%. Model performance degrades based on the number of rival values rather than total token count.