Research on Mixture-of-Experts (MoE) pruning shows that renormalizing survivor gates is critical for maintaining performance. The study compares various replacement operators on OLMoE and DeepSeek-V2-Lite, finding that gate scaling significantly impacts routing efficacy.
HOW THIS AFFECTS YOU
●
researcherWhen pruning MoE models, pay close attention to the gate scale of the remaining experts.