Self-Distillation Enables Continual Learning [PDF]

MIT/ETH researchers propose self-distillation as a continual learning mechanism for foundation models, submitted to arXiv Jan 2026. The method uses on-policy RL combined with self-distillation to prevent catastrophic forgetting without replay buffers. Practitioners training LLMs on sequential tasks or domains should watch this — it targets a core unsolved problem in production ML pipelines.