Early Data Exposure Improves Robustness to Subsequent Fine-Tuning

Studies how upstream training choices affect robustness of post-trained capabilities to downstream fine-tuning across 135M and 1B parameter models, showing early data exposure during pretraining improves retention of target capabilities through subsequent fine-tuning.