SpawnLoRA Mitigates Subspace Contention in MoE+LoRA Fine-Tuning

Adding biomedical data to code datasets increases code perplexity in MoE+LoRA despite disjoint routing. The authors introduce SpawnLoRA, which dynamically adds gated sub-adapters inside MoE experts to prevent orthogonal domain gradients from competing within the same low-rank subspace.