Romanized Pretraining Improves Multilingual Model Cross-Lingual Transfer

A systematic study of multilingual pretraining shows that romanized text yields stronger cross-lingual transfer than native orthography, IPA, or script equalization. The performance advantage of romanization over standard text scales with model size across 467M to 1.03B parameters.