Simplified Sparse Attention via Gist Tokens

Researchers propose a simplified approach to sparse attention that eliminates the need for architectural modifications. By incorporating gist tokens and attention masks during pretraining, the model learns to condense important information into a smaller set of tokens. This results in a 30% reduction in inference cost for long-context sequences on the 1.3B parameter T5 model, with minimal impact on performance.