ALRA Improves Logit-Based Knowledge Distillation for Autoregressive Models

ALRA uses position-specific adaptive local relational alignment to distill knowledge from teacher to student models. By combining student proposals with teacher anchors and adjusting candidate token counts based on probability distribution, it improves the alignment of next-token distributions.