Semi-OPD Outperforms On-Policy Distillation in Accuracy and Speed
October 8, 2026
Research shows that Semi-OPD, which distills from offline student rollouts, often outperforms on-policy distillation (OPD) across 17 teacher-student pairs. The method achieved up to 13.6% higher accuracy and an 11.4x training speedup, particularly when teacher-student output overlap is low.
HOW THIS AFFECTS YOU
●
builderYou can significantly reduce training costs and time when distilling knowledge into smaller models.
●
researcherYou can optimize your distillation pipeline by choosing offline rollouts instead of on-policy sampling.