Numerical analysis reveals that ALiBi positional encoding's linear bias scaling can underflow floating-point precision, zeroing out attention weights and causing partial blindness in attention heads. This failure mode significantly impairs token retrieval in state-of-the-art pretrained models.
HOW THIS AFFECTS YOU
●
builderBe aware that models using ALiBi may suffer from unexpected token retrieval failures in production.
●
researcherYou should implement one of the four proposed training-time mitigations when using ALiBi-based architectures.