Scaling Laws for Mixture Pretraining Under Data Constraints

Studies scaling laws for mixture pretraining when target data is limited, quantifying the trade-off between underexposure and overfitting when combining scarce domain-specific data with abundant generic data.