Continually Post-training Small Models via Synthetic Self-distillation

SynthSelf enables continual post-training for small models by using synthetic data to mitigate distribution shifts. The method improves knowledge acquisition and retention by replacing direct in-context instruction-following with in-distribution rewrites, addressing the failure modes where smaller models typically suffer from performance degradation during self-distillation.