SynthSelf Enables Continual Learning for Small Models via Synthetic Self-Distillation

SynthSelf addresses knowledge forgetting in small models by using a minimally larger model from the same family to perform in-distribution rewrites. This method utilizes off-policy supervision and rejection sampling to improve capability acquisition without the high in-context learning requirements of standard self-distillation.