Token-Level KL Direction Dominates Distillation Dynamics over Rollout Policy
September 27, 2026
A systematic study across Llama3 and Qwen2.5 families reveals that token-level KL direction is a more significant driver of distillation success than the rollout policy. The research isolates these variables across scientific, medical, and arithmetic reasoning tasks.
HOW THIS AFFECTS YOU
●
builderYou can focus on KL direction rather than complex rollout policy adjustments to improve student model performance.
●
researcherThis clarifies which hyperparameter to prioritize when optimizing distillation for reasoning models.