Cohere 发布 North Mini Code Megakernel 推理服务引擎
Cohere 发布面向 North Mini Code 的 Megakernel 推理服务引擎,在单张 H100 上以 BF16 运行,端到端平均解码吞吐量比 vLLM 快 1.25× - 1.41×。该引擎支持连续批处理、分页注意力、流式输出、前缀缓存和工具调用,并提供 OpenAI 兼容接口;其代码已在 GitHub 开放。批大小为 1 时,Megakernel 达到 292 tok/s,为 vLLM 的 1.58×,但当前最大批大小为 8,且预填充仍由普通 PyTorch 内核运行并会暂停解码。
thank Bharat Venkitesh for providing technical support throughout this work. We thank Stephen Jones, Brian Pharris, Vinod Grover, Frederic Bastien, Hua Huang and Disha Mehra from Nvidia for various insightful discussions. We thank Zewen Shen for some helpful discussions on evaluation and numerical accuracy. The design builds on ideas from Hazy Research's Look Ma, No Bubbles! and uses ThunderKittens tile primitives. Thanks also to the authors and maintainers of FlashAttention, PyTorch, Transform
匹配字段正文


