[arXiv]score: 0.36

DivMoE Enables Stable Fine-Grained MoE Upcycling via Domain Initialization

October 9, 2026

DivMoE addresses routing collapse in fine-grained MoE upcycling by using domain-specialized expert initialization from multiple dense models. This prevents the accuracy drops observed in naive upcycling, where models like Qwen3-1.7B dropped from 50.2% to 23.2% accuracy.

HOW THIS AFFECTS YOU

●

builderYou can more effectively convert dense models into fine-grained MoE architectures without losing downstream accuracy.

●

researcherYou can mitigate routing collapse during sparse upcycling by leveraging cross-domain expert composition.

●

founderYou can reduce the massive compute costs of training MoEs from scratch by using more stable upcycling methods.

read original ↗arxiv.org

DAILY DIGEST

you don't check 9 sources — we do. one email every morning, read in 2 min. free. unsubscribe anytime. privacy

← back to feed