Replacing Attention with Iterative Autoencoder Refinement in Transformers

A new masked language modeling approach replaces standard attention with a stack of low-rank bottleneck autoencoders for context mixing. The method uses local, sequence-wide, and head-wise modules combined with a two-step iterative refinement process to improve reconstruction.