Training looped language models requires optimizing backward windows

Research into looped transformer architectures shows that the choice of backward window—the number of layer repetitions retaining gradients—impacts the trade-off between per-token learning efficiency and per-update computational cost. The optimal window size shifts as total training budgets increase.