RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably

Theoretical proof that Rotary Positional Embeddings lose locality bias and token relevance consistency as context length increases, becoming unpredictable in long-context Transformers.