Looped GPT-BERT Achieves High Performance with Fewer Parameters
September 10, 2026
Looped GPT-BERT utilizes depth-wise parameter sharing to trade computation for parameter count in small language models. On the BabyLM 2026 dataset, a 12.18M parameter model with 12 recurrent traversals achieved comparable performance to larger GPT-2 and GPT-BERT baselines.
HOW THIS AFFECTS YOU
●
builderYou may be able to deploy highly capable small models by using looped architectures to reduce memory footprint.
●
researcherYou can investigate recurrent architectures to improve efficiency in data-constrained training scenarios.