Mitigating Length Inflation via EOS Token Alignment in Distillation
September 16, 2026
On-policy distillation causes student models to generate excessive length when termination-token probabilities mismatch between teacher and student. Aligning decoding stopping sets or treating equivalent EOS tokens as shared semantic actions mitigates this inflation in Qwen, Llama, and Gemma models.
HOW THIS AFFECTS YOU
●
builderImplementing semantic EOS alignment can prevent unexpected token consumption and latency spikes in distilled models.
●
researcherYou can improve distillation efficiency by accounting for EOS token probability mismatches.