Interaction Sparsification Explains LLM Knowledge Distillation Efficacy

This research identifies that successful knowledge distillation works by sparsifying nonlinear interactions within the student model. Performance differences between distillation methods are directly tied to how effectively they preserve complex input variable interactions while suppressing others.