Sparse Attention Accelerates Long-Context Decoding Without Retraining

A new sparse attention strategy accelerates long-context decoding in hybrid LLMs by combining sliding windows, pinned prefixes, and a fixed budget of relevant tokens. Testing on models up to 397B parameters shows decoding speeds improve by up to 2.22x for 256k contexts without losing accuracy.