Bilingual Pretraining Causes Hidden-State Mismatch in Decoder-Only Models
August 28, 2026
Pretraining 310M-parameter models on bilingual data reveals that while token embeddings align after vocabulary alignment, deeper hidden states remain distinct. This mismatch grows through middle transformer layers, indicating that language-conditioned effects arise from contextual processing rather than input representations.
HOW THIS AFFECTS YOU
●
researcherDo not assume that aligning embedding spaces ensures comparable cross-lingual representations in the model's deeper layers.