EPnG: Adaptive MoE Fine-tuning Updating Less Than 0.72% of Parameters
July 3, 2026
EPnG adapts LoRA capacity to Mixture-of-Experts (MoE) routing dynamics by pruning under-utilized experts and growing high-importance ones. In tests on OLMoE and Qwen1.5-MoE, it matches full fine-tuning performance while updating only 0.55% to 0.72% of parameters.
HOW THIS AFFECTS YOU
●
builderYou can achieve full fine-tuning performance on MoE models with up to 180x fewer parameter updates.
●
researcherThis method provides a way to align PEFT with router gate probabilities for better MoE adaptation.