Meta and MIT Paper: Reusing Transformer Layers in a Loop Can Outperform Standard Models
A new paper from Meta and MIT finds that reusing Transformer layers in a loop allows models to gain greater depth without adding parameters, and outperforms standard models. The research trained dense models of up to 1B parameters and up to 155B tokens, finding that looping the entire stack yields the best quality at a fixed scale, while looping intermediate layers gives the highest returns when training compute is tight. Each loop adapter adds only 1.1%–4.3% parameters, whereas doubling depth requires an increase of 51.4%.
Body
New Meta and MIT paper finds reusing a transformer's layers in loops can beat a plain model.
Looping runs the same layers several times, giving a model more depth without more parameters. But each loop costs compute and normally runs in sequence for every token.
They trained dense models up to 1B parameters on up to 155B tokens, testing where to loop and how many times. They also added tiny per-loop adapters and repaired a parallel looping method so loops can run at once during decoding.
Looping the whole stack gave the best quality at a fixed size, while looping middle layers paid off best when training compute was tight. Per-loop adapters added only 1.1 to 4.3% more parameters, versus 51.4% for doubling depth, and helped more as loops increased.