Mixture-of-Experts Models Overfit Faster to Repeated Training Data

Mixture-of-Experts (MoE) architectures exhibit faster performance degradation under data repetition compared to dense models. In models ranging from 80M to 1B active parameters, MoEs show significant deterioration at 4x data repetition, whereas 80M dense models maintain performance up to 8x repetition.