让知识蒸馏便宜到足以大规模运行
Multiverse Computing通过缓存教师模型每个位置的top-100 logits,并把输出投影融合进分块KL loss,大幅降低大语言模型离线知识蒸馏的显存成本。32K token基准中,峰值显存从85.2 GiB降至5.45 GiB;蒸馏GPT-OSS 20B时,GPU节点由4个减至1个,单步耗时从57.0秒降至12.23秒。该分块loss实现已开源。
rid and holds onto all of it, the other builds and discards one slice at a time, so memory never grows beyond a single chunk. We have open-sourced the chunked-loss implementation: github.com/CompactifAI/Full-Chunked-KL-Loss What this changes in practice The table below puts all four setups head to head: online distillation, and the three offline loss implementations just described. Comparing them on a single H200 GPU with Llama 3.1 8B Instruct as teacher and a 3.2B Llama model as student at an 8
匹配字段正文







