CuraWeb: Joint Optimization of Quality, Redundancy, and Diversity for Web-Scale Pretraining Data

CuraWeb optimizes for quality, redundancy, and diversity simultaneously using a multi-objective sampler and hybrid n-gram/semantic deduplication. This framework constructs a 2T-token English corpus from Common Crawl by balancing model-driven cleaning with distributional breadth to prevent the loss of long-tail knowledge common in single-objective pipelines.