Generalization Gaps in On-Policy Distillation for RL Consolidation
November 2, 2026
Analysis shows that on-policy distillation (OPD) fails to fully transfer teacher generalization capabilities. While in-distribution scores remain high, out-of-distribution scores drop significantly in mathematical reasoning and coding tasks.
HOW THIS AFFECTS YOU
●
researcherBe cautious when using OPD for post-RL consolidation; it may mask a loss in OOD reasoning capabilities.
●
policyThis highlights potential reliability issues in models distilled for complex reasoning tasks.