RAT+ Exponential Memory Boosts Sparse KV-Cache Methods on Long-Context Tasks

Adding exponentially decaying recurrence (RAT+) to sparse attention methods Quest, MoBA, and SnapKV consistently improves accuracy on eight needle-in-a-haystack tasks across sparse KV budgets. Gains validated on both RAT+ checkpoints and OLMo2-7B continued-pretrained for 10B tokens with the memory module added.