Bilingual pre-training outperforms hyperparameter tuning in low-resource settings

Mixing high-resource auxiliary language data during pre-training is more effective for low-resource languages than aggressive hyperparameter tuning. This approach expands the training distribution to prevent degradation caused by frequent data repetition.