Survey of LLM Attention Mechanisms and Contextual Memory Trade-offs

Dense self-attention scales quadratically with sequence length, creating bottlenecks in prefill costs and KV cache memory. This analysis categorizes emerging alternatives—including sparse access, recurrent states, and memory compression—across five dimensions of memory representation, update, access, readout, and integration.