Representation Geometry Regularization (RGR) mitigates capability loss during on-policy self-distillation by maintaining cluster compactness and separation in hidden representations. This prevents optimization from degrading previously acquired skills while improving overall performance.
HOW THIS AFFECTS YOU
●
researcherYou can apply RGR to stabilize performance during iterative self-improvement training loops.