Reducing Distillation Cost by Using Sparse Vocabulary Support
November 13, 2026
Distilling from a teacher's top 0.01% vocabulary fraction shows no MMLU-Pro degradation in Qwen3 4B to 1.7B transfers. This approach reduces the computational expense of Forward-KL distillation by avoiding full-vocabulary score computation.
HOW THIS AFFECTS YOU
●
builderYou can train smaller student models much faster and cheaper by targeting only the top vocabulary fraction.
●
researcherThis validates that sparse support can maintain performance in reasoning-heavy distillation tasks.