Impact of Expert Deactivation Methods on MoE Inference Perplexity

Inference-time expert deactivation significantly alters model perplexity depending on whether renormalization or unnormalized re-routing is used. The optimal deactivation method is tied to the specific weight rule used during the model's initial training.