LastOPD addresses latent collapse in on-policy distillation

Latent supervision in on-policy distillation of Qwen3 models can lead to performance collapse, where MATH-500 accuracy drops from 46 to 11 despite improving alignment metrics. The research identifies a mismatch in how latent signals are applied across layers during the distillation of 4B and 8B models into 1.7B bases.