Direct On-Policy Distillation for Weak-to-Strong Generalization
July 7, 2026
Direct-OPD enables transferring RL-induced policy shifts from a smaller, cheaper teacher model to a larger target model. By distilling the shift in policy rather than the final policy, it avoids inheriting the limitations of the smaller teacher model.
HOW THIS AFFECTS YOU
●
builderYou can reduce post-training costs by performing RL on small models and distilling the gains to larger ones.
●
researcherThis offers a more efficient path for weak-to-strong generalization than standard policy distillation.