Impact of Positional Encoding on Transformer Distance Generalization

Investigation into distance generalization shows how transformers perform when inter-token distances change between training and inference. The study evaluates whether RoPE and ALiBi improve distance resolution compared to NoPE and examines how training data diversity affects the ability to generalize to unseen delays.