Nathan Lambert says DeepSeek processes 150T tokens per day on a single cluster, more than the combined open-source model inference volume on OpenRouter, Fireworks, Together, and BaseTen. He also posted a reconstructed comparison chart of his own based on these companies' inference volume reports, and noted that this figure corresponds to only a single cluster; it remains unclear how many clusters DeepSeek actually has.
MiniMax announced a partnership with Inferact to jointly advance production inference for MiniMax models, with the related engineering improvements contributed back to the vLLM project. MiniMax also thanked the Inferact team for supporting M3.1 Flash in production from day one.
Cohere has launched confidential computing capabilities on Model Vault, currently in public beta, running on NVIDIA GPUs on Microsoft Azure. The feature combines confidential CPU and GPU execution, OHTTP end-to-end encryption of request content, and remote attestation, allowing customers to verify the target environment before releasing sensitive data. Cohere plans to open-source the confidential inference stack and advance support for NVIDIA Blackwell and B200 GPUs.
MiniMax officially responded to SGLang adding MiniMax H3 to the TPU support list of SGLang-TorchTPU, and said it looks forward to more open-source work on efficient video inference. SGLang said that SGLang-TorchTPU was built jointly by SGLang, the Google TPU team, and the Meta PyTorch team, has been used for production inference at Meta, and supports models including Muse Spark Open, GLM-5.3 Flash, Kimi K3, Ling 3.1 Flash, and MiniMax H3.
VocabX selects tokens by cost rather than frequency and simultaneously rebuilds subword segmentation and multi-word merging, compressing the vocabulary of domain-specific small models to a smaller scale and reducing inference costs by 77% in real-world applications. Taking a scoring model as an example, generating one token costs 7.4 times as much as reading one token, and the output-layer vocabulary shrinks from about 152,000 rows to about 65,000 rows. This method targets the memory bandwidth bottleneck of small models such as Qwen3-4B at medium batch sizes, whereas methods such as FVT, MWT, and in-place BPE select tokens only by frequency and address only a single level.
VocabX selects tokens by cost rather than frequency and simultaneously rebuilds subword segmentation and multi-word merging, compressing the vocabulary of domain-specific small models to a smaller scale and reducing inference costs by 77% in real-world applications. Taking a scoring model as an example, generating one token costs 7.4 times as much as reading one token, and the output-layer vocabulary shrinks from about 152,000 rows to about 65,000 rows. This method targets the memory bandwidth b
DeepSeek has open-sourced its internal inference framework Deepinfer on GitHub, with the goal of reducing the compute and energy costs of deploying models on your own. During the inference stage, the framework distributes load more evenly across available processor cores, uses speculative decoding to speed up responses, and improves the structure of results before output; it currently supports only DeepSeek-V4.1-Flash, with more of its own models to be opened up later. On the hardware side, it requires Nvidia GPUs with the SM100 architecture or Huawei Ascend 950 NPUs, so it is not aimed at individual users.
DeepSeek has open-sourced its internal inference framework Deepinfer on GitHub, with the goal of reducing the compute and energy costs of deploying models on your own. During the inference stage, the framework distributes load more evenly across available processor cores, uses speculative decoding to speed up responses, and improves the structure of results before output; it currently supports only DeepSeek-V4.1-Flash, with more of its own models to be opened up later. On the hardware side, it r
DeepSeek has open-sourced its internal inference framework DeepInfer, which it says powers the high-throughput, low-latency inference of DeepSeek-V4.1-Flash. The framework supports both Nvidia and Huawei Ascend hardware, and offers capabilities such as distributed inference, speculative decoding, SWA Bounded Replay, and structured output. DeepSeek also released deepseek-recipe, a production-grade tokenization component fully aligned with the official API.
DeepSeek has open-sourced DeepInfer, its self-developed inference framework that supports DeepSeek-V4.1-Flash, and published production data: a single cluster processes 155 trillion tokens per day, 96% input cache hit rate, 400ms first-token latency, and 230 tokens/s generation speed. The framework is customized based on vLLM and supports distributed inference, speculative decoding, SWA Bounded Replay, and structured output, with target platforms NVIDIA SM100 and Huawei Ascend 950. DeepSeek also states that components such as KV cache management and load balancing are not yet fully open-sourced, while the vLLM team says that under a matching GB200 configuration, decoding throughput is within 6% of the results published by DeepInfer.
DeepSeek has open-sourced DeepInfer, its self-developed inference framework that supports DeepSeek-V4.1-Flash, and published production data: a single cluster processes 155 trillion tokens per day, 96% input cache hit rate, 400ms first-token latency, and 230 tokens/s generation speed. The framework is customized based on vLLM and supports distributed inference, speculative decoding, SWA Bounded Replay, and structured output, with target platforms NVIDIA SM100 and Huawei Ascend 950. DeepSeek also
DeepSeek has open-sourced DeepInfer, the production inference framework supporting V4.1 Flash, and published production data including 155 trillion tokens processed per day on a single cluster, a 96% input cache hit rate, an average 400ms time to first token, and 230 tokens/s generation. The framework is customized based on vLLM and uses the Apache 2.0 license, supports distributed inference, speculative decoding, SWA Bounded Replay and structured output, and targets NVIDIA SM100 and Huawei Ascend 950. DeepSeek says components such as KV cache management and load balancing are not fully open-sourced because they are coupled with internal infrastructure, while the vLLM team reports that under matched GB200 configurations, decoding throughput differs from DeepInfer's published results by less than 6%.
DeepSeek has open-sourced DeepInfer, the production inference framework supporting V4.1 Flash, and published production data including 155 trillion tokens processed per day on a single cluster, a 96% input cache hit rate, an average 400ms time to first token, and 230 tokens/s generation. The framework is customized based on vLLM and uses the Apache 2.0 license, supports distributed inference, speculative decoding, SWA Bounded Replay and structured output, and targets NVIDIA SM100 and Huawei Asce
