Sparse Delta Memory (SDM) scales the hidden state of gated linear RNNs using a sparse addressing scheme to improve long-context recall. By replacing dense key-value outer products with sparse reads and writes to a large memory, it achieves higher capacity under isoFLOP constraints compared to standard linear attention.
HOW THIS AFFECTS YOU
●
researcherYou can scale linear RNN state capacity for long-context tasks without the linear FLOP increase usually required.