Hybrid Attention Architectures Impact Multilingual LLM Performance

This study investigates how hybrid attention mechanisms, which combine softmax and recurrent attention, affect the multilinguality and inductive biases of LLMs. The analysis reveals that the ordering of recurrent and full attention components can alter cross-lingual representation patterns.