DeepSeek 开源推理框架 DeepInfer,单集群日处理 155 万亿 token
DeepSeek 开源了支撑 DeepSeek-V4.1-Flash 的自研推理框架 DeepInfer,并公布单集群每天处理 155 万亿 token、96% 输入缓存命中、首 token 延迟 400ms、生成速度 230 tokens/s 的生产数据。该框架基于 vLLM 定制,支持分布式推理、投机解码、SWA Bounded Replay 和结构化输出,目标平台为 NVIDIA SM100 与华为昇腾 950。DeepSeek 同时说明 KV cache 管理和负载均衡等组件尚未完全开源,vLLM 团队则称在 GB200 匹配配置下解码吞吐与 DeepInfer 公布结果相差 6% 以内。
eek - DeepInfer - (Activity: 313): The image is a mobile screenshot of DeepSeek’s X announcement that DeepInfer, its in-house inference framework, is now fully open source: image, GitHub. The technically relevant claim from the linked/source discussion is that DeepInfer powers DeepSeek V4.1 Flash at scale, with reported single-cluster throughput of 155 trillion tokens/day and revenue of $38 per card per hour at off-peak pricing, emphasizing high-throughput, low-latency large-scale inference rath
匹配字段正文



