StickyMoE Reduces MoE Expert Switching by 60% During Inference

StickyMoE introduces a differentiable routing consistency loss to discourage abrupt expert switches in Mixture-of-Experts models. This method reduces expert switching by up to 60% with less than 4% perplexity increase, improving memory efficiency on edge devices.