Slow Disagreement Decay in Masked Text Reconstruction Models
October 12, 2026
Increasing corpus size from 512 to 2,048 chunks only reduces predictor disagreement by 5--40%, failing to meet the expected 75% reduction from 1/n scaling. The discrepancy is attributed to the smoothing of rarely seen contexts in the GPT-2 vocabulary.
HOW THIS AFFECTS YOU
●
researcherThis suggests that corpus scaling alone may not resolve predictor variance as quickly as theoretical models predict.