Compute-Efficient Hyperparameter Transfer for Large-Scale MoE Training

This framework estimates optimal learning rates for MoE models by transferring parameters across scaled widths using MuP adaptation and Muon optimization. It further enables extrapolation to trillion-token horizons through a predictive scaling law, reducing the need for expensive hyperparameter sweeps.