Meta 与 MIT 论文：循环复用 Transformer 层可胜过普通模型
New Meta and MIT paper finds reusing a transformer's layers in loops can beat a plain model.
Meta 与 MIT 新论文发现，循环复用 Transformer 层能让模型在不增加参数的情况下获得更深深度，并优于普通模型。研究训练了最高 1B 参数、最多 155B token 的稠密模型，发现循环整个堆栈在固定规模下质量最佳，训练算力紧张时循环中间层收益最高。每循环适配器仅增加 1.1%–4.3% 参数，而翻倍深度需增加 51.4%。
正文
New Meta and MIT paper finds reusing a transformer's layers in loops can beat a plain model.
Looping runs the same layers several times, giving a model more depth without more parameters. But each loop costs compute and normally runs in sequence for every token.
They trained dense models up to 1B parameters on up to 155B tokens, testing where to loop and how many times. They also added tiny per-loop adapters and repaired a parallel looping method so loops can run at once during decoding.
Looping the whole stack gave the best quality at a fixed size, while looping middle layers paid off best when training compute was tight. Per-loop adapters added only 1.1 to 4.3% more parameters, versus 51.4% for doubling depth, and helped more as loops increased.