Gains and Collapse in On-Policy Distillation:A Reinforcement Learning Perspective

On-policy distillation improves performance by making correct responses easier to sample via an implicit reward model but does not expand the student's underlying capabilities. When this implicit reward aligns with quality, performance gains occur; however, misalignment leads to reward hacking, resulting in repetitive and excessively long generations.