LMSYS explains in detail how HiCache adapts to multi-parallelism and distributed inference
The LMSYS blog introduces how HiCache is implemented in context parallelism, pipeline parallelism, speculative decoding, unified KV layout, and P/D disaggregation scenarios, with the core being to keep cache decisions consistent across ranks. The iFlytek Spark MaaS team used HiCache plus external storage to serve GLM-5.1/5.2 CodingPlan workloads, raising the cache hit rate from about 30% to 95% and reducing median TTFT to under 3 seconds. In CP sharding tests, the GPU token hit rate increased by 40.48 percentage points, the host memory serving token volume dropped by 81.2%, and the average prefill queueing time decreased by 56.2%.

The LMSYS blog introduces how HiCache is implemented in context parallelism, pipeline parallelism, speculative decoding, unified KV layout, and P/D disaggregation scenarios, with the core being to keep cache decisions consistent across ranks. The iFlytek Spark MaaS team used HiCache plus external storage to serve GLM-5.1/5.2 CodingPlan workloads, raising the cache hit rate from about 30% to 95% and reducing median TTFT to under 3 seconds. In CP sharding tests, the GPU token hit rate increased by
Matched fieldsTitle






