Host-Aware Web Crawling Strategies for LLM Pretraining

Research indicates that roughly 50% of document quality variation occurs between hosts, suggesting that sampling a few pages can estimate a host's overall utility. Simulations show that using harmonic centrality to guide crawl allocation provides limited gains over random ordering in data collection.