Directional Label Distillation for More Accurate LLM Knowledge Transfer

Directional label distillation improves student model accuracy by using frozen teachers to score candidate answers in directions they already know, rather than the requested direction. This approach mitigates the propagation of directional knowledge gaps during the distillation process.