PMOPD Mitigates Capability Seesaw in Multi-Teacher Distillation
September 27, 2026
PMOPD addresses cross-task interference in multi-teacher distillation by identifying that parameter updates concentrate in task-specific low-dimensional subspaces. This geometric insight allows for better control over the 'capability seesaw' effect where improving one skill suppresses another.
HOW THIS AFFECTS YOU
●
builderThis method provides a way to integrate multiple specialized capabilities into a single model without losing performance in existing domains.
●
researcherYou can use subspace protection to stabilize multi-objective distillation training.