Reducing MoE All-to-All Overhead via Correlated Expert Placement

To mitigate the 45-60% training step overhead caused by all-to-all collectives in Mixture-of-Experts (MoE), this method exploits correlated router patterns. By leveraging the fact that token-expert assignments are correlated across layers, the method enables more efficient expert placement and token shuffling.