Comparative Analysis of Linear Attention Architectures via Recurrent-Memory Notation

This study evaluates DeltaNet, Gated DeltaNet, Kimi Delta Attention, and Gated DeltaNet-2 using a unified recurrent-memory notation. Benchmarks on 350M to 3B parameter models demonstrate how varying memory decay and write control mechanisms impact training throughput and expressivity relative to softmax attention.