Analyzing Transformer Layers as Distributed Memory Stores

Transformer architectures function as hierarchical memory systems where linear maps, MLPs, and self-attention layers serve distinct storage roles. The capacity for accurate recall is constrained by the number of stored memories, creating a trade-off between density and retrieval fidelity.