Muon-RT: Momentum-Free LLM Self-Distillation via Teacher Trajectories
October 29, 2026
Muon-RT eliminates the need for separate momentum buffers in LLM self-distillation by reusing the teacher-student displacement. This approach reduces persistent training memory while maintaining the benefits of exponentially weighted history for hidden weight matrices.
HOW THIS AFFECTS YOU
●
researcherYou can reduce training memory overhead during self-distillation by reusing teacher trajectories.