On October 29, 2026, Ant Ling, in collaboration with the NVIDIA AI team, optimized the inference of the open-weight model Ling-3.0-flash on a single NVIDIA DGX Spark using three techniques. On 1,319 GSM8K problems, decoding throughput increased from 37.81 tok/s to 53.89 tok/s (+42.5%), and average latency dropped from 15.21s to 11.49s (-24.5%), with no statistically detectable change in accuracy. There are currently no further developments.
On October 30, 2026, Baseten became Postman's recommended inference partner for enterprise agents, responsible for providing production inference and post-training support for agents built through the Postman Agent Stack. In an engineering blog post, Baseten said that its seven model APIs released in 2026 all ranked in the top two in Artificial Analysis's throughput and latency rankings, with Beam 1.0, DeepSeek V4.1 Flash and others taking first place. The blog post also mentioned that Harvey used Baseten to post-train a 27B open-source model, matching closed-source frontier models on legal agent benchmarks, while the latter cost about $50 per task. There are currently no subsequent developments.
On October 30, 2026, Baseten became Postman's recommended inference partner for enterprise agents, responsible for providing production inference and post-training support for agents built through the Postman Agent Stack. In an engineering blog post, Baseten said that its seven model APIs released in 2026 all ranked in the top two in Artificial Analysis's throughput and latency rankings, with Beam 1.0, DeepSeek V4.1 Flash and others taking first place. The blog post also mentioned that Harvey us
On October 30, 2026, Baseten became Postman's recommended inference partner for enterprise agents, responsible for providing production inference and post-training support for agents built through the Postman Agent Stack. In an engineering blog post, Baseten said that its seven model APIs released in 2026 all ranked in the top two in Artificial Analysis's throughput and latency rankings, with Beam 1.0, DeepSeek V4.1 Flash and others taking first place. The blog post also mentioned that Harvey used Baseten to post-train a 27B open-source model, matching closed-source frontier models on legal agent benchmarks, while the latter cost about $50 per task. There are currently no subsequent developments.
On October 29, 2026, DeepSeek open-sourced the inference engine DeepInfer on GitHub; this engine is the inference engine behind DeepSeek-V4.1-Flash. Leiphone reported the same day that DeepInfer includes technologies such as MoE expert parallelism, asynchronous batch scheduling, custom kernels, and speculative decoding, and disclosed that a single cluster needs to process approximately 155 trillion tokens per day. Its Zero-Overhead Batch Scheduler lets the CPU schedule computation one round in advance, overlapping with GPU forward passes, and handles cross-round state misalignment through total_inflight_tokens accounting, Remap mapping, and DeferredRelease deferred reclamation. Currently, only offline inference capabilities are open; components such as online KV Cache storage and cross-replica load balancing have not yet been fully disclosed, and the official team has not separately published end-to-end speedup data for this scheduling mechanism relative to synchronous solutions.
On October 28, 2026, the Google Developers Blog announced that LiteRT in Google Play services has received its biggest upgrade yet: GPU inference based on ML Drift is about 1.4 times faster than the old GPU delegate and is provided through the new CompiledModel API. After switching to this runtime, app size shrinks by about 8 MB, because the runtime is distributed uniformly with Play services and is no longer packaged into the APK. Currently, more than 100,000 Android apps already bundle LiteRT or TensorFlow Lite. The solution supports Android 7 and above, Wear OS 4+, and Android Automotive OS (Android 11+), while NPU support is still under development.
On October 28, 2026, the Google Developers Blog announced that LiteRT in Google Play services has received its biggest upgrade yet: GPU inference based on ML Drift is about 1.4 times faster than the old GPU delegate and is provided through the new CompiledModel API. After switching to this runtime, app size shrinks by about 8 MB, because the runtime is distributed uniformly with Play services and is no longer packaged into the APK. Currently, more than 100,000 Android apps already bundle LiteRT
On October 27, 2026, OpenRouter announced that it will conduct child sexual abuse material (CSAM) screening on all inference requests during request processing. According to its announcement, the screening includes comparing content against hashes of known CSAM, and will also retain suspected content that does not match a hash but is flagged by classifiers; content flagged as suspected CSAM may be retained for manual review and reported and preserved in accordance with the law. The announcement also states that this screening applies equally to requests covered by OpenRouter's zero data retention (ZDR) policy, and that ZDR does not exempt the obligations to report and preserve data in accordance with the law. There are currently no subsequent developments.
On October 27, 2026, OpenRouter announced that it will conduct child sexual abuse material (CSAM) screening on all inference requests during request processing. According to its announcement, the screening includes comparing content against hashes of known CSAM, and will also retain suspected content that does not match a hash but is flagged by classifiers; content flagged as suspected CSAM may be retained for manual review and reported and preserved in accordance with the law. The announcement
On October 27, 2026, Google launched a replaceable reasoning format for A2UI, with A2UI Express becoming the new default format. According to the Google Developers Blog, Express can reduce latency by up to 3x and reduce the tokens needed to generate comparable UI by up to 3x; when used with decision models such as Jev for layout decisions, the latency reduction can reach 80x. The format has landed in the Python agent SDK (a2ui-agent-sdk 0.9.0), the TypeScript SDK (@a2ui/agent), and the Dart SDK (a2ui_agent 0.0.1-wip004), and developers can switch formats with a single parameter when creating a generator. There are currently no reports of further developments.
On October 27, 2026, Google launched a replaceable reasoning format for A2UI, with A2UI Express becoming the new default format. According to the Google Developers Blog, Express can reduce latency by up to 3x and reduce the tokens needed to generate comparable UI by up to 3x; when used with decision models such as Jev for layout decisions, the latency reduction can reach 80x. The format has landed in the Python agent SDK (a2ui-agent-sdk 0.9.0), the TypeScript SDK (@a2ui/agent), and the Dart SDK
On October 28, 2026, the Google Developers Blog announced that LiteRT in Google Play services has received its biggest upgrade yet: GPU inference based on ML Drift is about 1.4 times faster than the old GPU delegate and is provided through the new CompiledModel API. After switching to this runtime, app size shrinks by about 8 MB, because the runtime is distributed uniformly with Play services and is no longer packaged into the APK. Currently, more than 100,000 Android apps already bundle LiteRT or TensorFlow Lite. The solution supports Android 7 and above, Wear OS 4+, and Android Automotive OS (Android 11+), while NPU support is still under development.
On October 27, 2026, OpenRouter announced that it will conduct child sexual abuse material (CSAM) screening on all inference requests during request processing. According to its announcement, the screening includes comparing content against hashes of known CSAM, and will also retain suspected content that does not match a hash but is flagged by classifiers; content flagged as suspected CSAM may be retained for manual review and reported and preserved in accordance with the law. The announcement also states that this screening applies equally to requests covered by OpenRouter's zero data retention (ZDR) policy, and that ZDR does not exempt the obligations to report and preserve data in accordance with the law. There are currently no subsequent developments.
On October 27, 2026, Google launched a replaceable reasoning format for A2UI, with A2UI Express becoming the new default format. According to the Google Developers Blog, Express can reduce latency by up to 3x and reduce the tokens needed to generate comparable UI by up to 3x; when used with decision models such as Jev for layout decisions, the latency reduction can reach 80x. The format has landed in the Python agent SDK (a2ui-agent-sdk 0.9.0), the TypeScript SDK (@a2ui/agent), and the Dart SDK (a2ui_agent 0.0.1-wip004), and developers can switch formats with a single parameter when creating a generator. There are currently no reports of further developments.
On October 27, 2026, Redwood Research published a post on its blog arguing that AI's conceptual reasoning capabilities should be accelerated differentially—that is, tasks akin to philosophy that lack verifiable standards and rely on a priori argumentation. The organization holds that such capabilities are crucial to safety work such as alignment, yet are neglected by AI companies because of their low value to AI R&D and commercial applications, and are currently underinvested. Its preliminary work shows that basic steering methods such as few-shot prompting and small-scale training can improve models' scores on the main conceptual reasoning benchmark by an amount equivalent to 3 to 12 months of frontier progress.
On October 27, 2026, Redwood Research published a post on its blog arguing that AI's conceptual reasoning capabilities should be accelerated differentially—that is, tasks akin to philosophy that lack verifiable standards and rely on a priori argumentation. The organization holds that such capabilities are crucial to safety work such as alignment, yet are neglected by AI companies because of their low value to AI R&D and commercial applications, and are currently underinvested. Its preliminary work shows that basic steering methods such as few-shot prompting and small-scale training can improve models' scores on the main conceptual reasoning benchmark by an amount equivalent to 3 to 12 months of frontier progress.
On October 27, 2026, Redwood Research published a post on its blog arguing that AI's conceptual reasoning capabilities should be accelerated differentially—that is, tasks akin to philosophy that lack verifiable standards and rely on a priori argumentation. The organization holds that such capabilities are crucial to safety work such as alignment, yet are neglected by AI companies because of their low value to AI R&D and commercial applications, and are currently underinvested. Its preliminary wo
On October 20, 2026, DeepSeek open-sourced its internal inference framework DeepInfer. According to Yuchen Jin, citing data disclosed by DeepSeek, a single cluster processes 155 trillion tokens per day and generates $38 in revenue per GPU-hour (off-peak pricing); DeepInfer supports both NVIDIA Blackwell and Huawei Ascend. A subsequent 36Kr report stated that its accompanying open-source deepseek-recipe supports NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. On October 21, swyx, citing Latent.Space data, said a single cluster processes 155T tokens per day with a 96% input cache hit rate, and argued that the lifetime value of each sold GPU has risen by about 30%. On October 24, Nathan Lambert said DeepSeek's single cluster processes 150T tokens per day, exceeding the combined open-source model inference volume of OpenRouter, Fireworks, Together, and BaseTen, and noted that the actual number of clusters is unknown; on the same day, LMSYS published a same-hardware benchmark comparison, stating that SGLang meets or exceeds DeepInfer on DeepInfer's own benchmark, with prefill throughput up to 1.04x and decode step length shortened by 1.1%–7.4% on 64x GB300.
On October 20, 2026, DeepSeek open-sourced its internal inference framework DeepInfer. According to Yuchen Jin, citing data disclosed by DeepSeek, a single cluster processes 155 trillion tokens per day and generates $38 in revenue per GPU-hour (off-peak pricing); DeepInfer supports both NVIDIA Blackwell and Huawei Ascend. A subsequent 36Kr report stated that its accompanying open-source deepseek-recipe supports NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. On October 21, swyx, citing Latent.Spac
The Lenovo team tested the inference performance of a compute-storage fusion pooled system. Xiao Limin, Chief Scientist of Lenovo Group, proposed a compute-storage fusion pooled system that combines memory pooling with near-memory computing, targeting two major pain points: insufficient KV Cache capacity and high GPU memory access overhead. Tests show that memory pool offloading reduces first-token latency by 3.5x, KV cross-instance sharing achieves up to 12~28x speedup compared with full recomputation, offloading attention computation reduces decoding latency by 2.5x, and end-to-end throughput improves by 6~20x as context length grows. The team's next step is to evolve from G2.5 to G1.5 with the help of supernode scale-up interconnect.
The Lenovo team tested the inference performance of a compute-storage fusion pooled system. Xiao Limin, Chief Scientist of Lenovo Group, proposed a compute-storage fusion pooled system that combines memory pooling with near-memory computing, targeting two major pain points: insufficient KV Cache capacity and high GPU memory access overhead. Tests show that memory pool offloading reduces first-token latency by 3.5x, KV cross-instance sharing achieves up to 12 28x speedup compared with full recomp
The Lenovo team tested the inference performance of a compute-storage fusion pooled system. Xiao Limin, Chief Scientist of Lenovo Group, proposed a compute-storage fusion pooled system that combines memory pooling with near-memory computing, targeting two major pain points: insufficient KV Cache capacity and high GPU memory access overhead. Tests show that memory pool offloading reduces first-token latency by 3.5x, KV cross-instance sharing achieves up to 12~28x speedup compared with full recomputation, offloading attention computation reduces decoding latency by 2.5x, and end-to-end throughput improves by 6~20x as context length grows. The team's next step is to evolve from G2.5 to G1.5 with the help of supernode scale-up interconnect.
On October 23, 2026, MiniMax announced a partnership with Inferact to jointly advance production inference for MiniMax models. MiniMax stated that the relevant engineering improvements have been contributed back to the vLLM project, and thanked the Inferact team for supporting M3.1 Flash in the production environment from day one. Currently, this partnership was officially announced by MiniMax, and no further progress has been disclosed.
On October 23, 2026, MiniMax announced a partnership with Inferact to jointly advance production inference for MiniMax models. MiniMax stated that the relevant engineering improvements have been contributed back to the vLLM project, and thanked the Inferact team for supporting M3.1 Flash in the production environment from day one. Currently, this partnership was officially announced by MiniMax, and no further progress has been disclosed.
MiniMax responded to SGLang including MiniMax H3 in the TPU support list of SGLang-TorchTPU, expressing thanks and saying it looks forward to more open-source work around efficient video inference. SGLang said that SGLang-TorchTPU was jointly built by SGLang, the Google TPU team, and the Meta PyTorch team, has been used for Meta's production inference, and supports models including Muse Spark Open, GLM-5.3 Flash, Kimi K3, Ling 3.1 Flash, and MiniMax H3. There are currently no facts about subsequent developments.
MiniMax responded to SGLang including MiniMax H3 in the TPU support list of SGLang-TorchTPU, expressing thanks and saying it looks forward to more open-source work around efficient video inference. SGLang said that SGLang-TorchTPU was jointly built by SGLang, the Google TPU team, and the Meta PyTorch team, has been used for Meta's production inference, and supports models including Muse Spark Open, GLM-5.3 Flash, Kimi K3, Ling 3.1 Flash, and MiniMax H3. There are currently no facts about subsequ