UpLoop Training Method for Efficient Looped Transformers via Upcycling
November 2, 2026
UpLoop trains looped Transformers using an upcycling schedule that matches the training FLOPs and parameter count of standard dense models. At 1.3B parameters, this method increased average downstream task performance from 22.99 to 27.52 by allowing deeper computation through shared weights.
HOW THIS AFFECTS YOU
●
builderYou can achieve better reasoning performance at fixed compute budgets by implementing looped architectures.
●
researcherThis offers a way to increase model depth and reasoning capabilities without increasing pretraining budget.