Competing Learning in Hybrid Recurrent-Attention Architectures

Research reveals a Competing Learning phenomenon where modern recurrence layers suppress softmax attention signals by learning retrieval functions faster. This leads to attention layers failing to learn retrieval capabilities in hybrid architectures under typical data distributions.