Full BPTT Preferred Over Truncated Backpropagation in Looped Transformers

A scaling study of nearly 600 training runs shows that while truncated backpropagation through time (BPTT) saves compute, it introduces biased gradients. The study's unified scaling law indicates that full BPTT is superior when forward and backward loop counts are high.