Expert Routing Specialization in Bilingual MoE Transformers
August 18, 2026
Analysis of English-German Mixture-of-Experts models shows that curriculum-based sequential training can actually result in lower category-dependent routing specialization than mixed-data training. Peak mutual information reached 0.2599 for mixed training versus 0.1148 for curriculum-trained models at layer 5.
HOW THIS AFFECTS YOU
●
researcherThis challenges the assumption that sequential language exposure improves linguistic specialization in MoE architectures.