Zyphra 研究:MoE 预训练中的专家耦合,将 all-to-all 通信时间最多降低 2.63 倍
Zyphra 研究发现 MoE 路由存在可利用的结构:同一 token 选中的专家高度相关,且一层中的专家选择能预测后续层的路由。基于此提出相关专家放置与 token 重排两种方法,不改变模型架构或路由决策,在 MI300X GPU 的 Megatron-LM 实验中把 all-to-all 通信时间最多降低 2.63 倍,端到端训练步时间最多提升 1.41 倍。

ame remote GPU, a dispatcher can send three copies of the token's hidden state—one for each expert. A deduplicating dispatcher instead sends the token to that GPU once, along with metadata indicating which local experts should process it. Correlated placement deliberately creates more opportunities for this deduplication. Figure 3: Correlated placement and deduplication. Baseline placement sends the same token to multiple destinations—or multiple times to the same destination. Correlated placeme
匹配字段正文



