On-policy self-distillation suffers performance degradation during training
November 11, 2026
Research shows that on-policy self-distillation (OPSD) fails during continued training due to distributional mismatch between the frozen teacher and the evolving student. This mismatch leads to substantial performance loss as training progresses.
HOW THIS AFFECTS YOU
●
researcherBe careful when using self-distillation for long training runs, as the model may eventually learn from incorrect teacher signals.