Understanding Mixture-of-Experts Language Models Beyond Expert Boundaries

Routing-Aware Magnitude Profiling (RAMP) reveals that domain specialization occurs at the neuron level rather than the expert level. Analysis of three open MoE families shows 58-74% overlap in domain-specific neuron sets within the same domain, compared to 15-24% across different domains, even as expert-level routing remains stable.