Recurrent Depth Enhances Text-Only Performance in Data-Limited Pretraining
November 5, 2026
Repeating a 12-layer Transformer four times exceeds a standard 48-layer model by 5.4 BLiMP points under a 100M-word budget. However, recurrence does not mitigate the 3.8-point BLiMP accuracy drop observed during multimodal training using frozen DINOv2 features.
HOW THIS AFFECTS YOU
●
researcherWeight reuse through recurrence is an effective way to scale compute during data-constrained pretraining, though it may not solve multimodal alignment issues.