Scaling Repeated-Data Pretraining: Batch Size, Regularization, and GPU-Time Tradeoffs

Optimizing repeated-data pretraining requires simultaneous tuning of batch size and epoch counts, as smaller batches provide implicit regularization that lowers validation loss. Practitioners can improve performance on finite datasets by implementing document repacking, weight tying, and positive embedding weight decay rather than relying on single-pass optimization recipes.