Streaming On-Policy Distillation Decouples Response and Update Clocks
November 18, 2026
Streaming OPD allows response completion, supervision delivery, and parameter updates to operate at different granularities. Distilling Qwen3-4B into Qwen3-0.6B using this method reduced the early accuracy gap from 4.55 to 2.42 points on GSM8K.
HOW THIS AFFECTS YOU
●
builderThis method helps bridge the accuracy gap faster when distilling large models into smaller ones.
●
researcherYou can improve distillation efficiency by decoupling the timing of response generation and parameter updates.