Repeated Small Datasets Enable Faster Training via Sampling Biases

Repeating smaller datasets can be more compute-efficient than training on larger datasets due to layer-wise growth enabled by sampling biases. This effect is consistent across architectures and optimizers, offering a proactive optimization strategy for reasoning tasks.