Multi-Head Attention Residuals Improve Information Propagation

MHAR relaxes the single additive residual stream in Transformers by allowing sublayers to attend to depth history via learned softmax routing. This block-diagonal approach adds zero parameters and negligible compute, enabling feature subspaces to access different depth histories.