Information-Efficiency Ratio Optimizes Sparse On-Policy Distillation
September 20, 2026
This method introduces the Information-Efficiency Ratio (IER) to improve gradient estimation in sparse on-policy distillation. By selecting tokens based on IER and usefulness scores, it reduces noise in the gradient updates during student-teacher training.
HOW THIS AFFECTS YOU
●
researcherYou can improve the efficiency of distillation by targeting tokens with higher signal-to-noise ratios.