On October 20, 2026, AWS introduced a new bidirectional KV cache transfer mechanism in vLLM: in subsequent turns, the decode node sends the already-computed KV cache back to the prefill node, avoiding recomputation of the conversation history. On a 1P1D deployment on Amazon EC2 P5en, this mechanism improves TTFT for Qwen3-32B by up to 1.6x and for zai-org/GLM-5-FP8 by up to 3.3x. The capability was initially validated on AWS Trainium clusters and contributed to vLLM as an accelerator-agnostic feature that requires coordination with a KV-cache-aware router for cross-turn reuse. There are currently no further developments.
On October 20, 2026, AWS introduced a new bidirectional KV cache transfer mechanism in vLLM: in subsequent turns, the decode node sends the already-computed KV cache back to the prefill node, avoiding recomputation of the conversation history. On a 1P1D deployment on Amazon EC2 P5en, this mechanism improves TTFT for Qwen3-32B by up to 1.6x and for zai-org/GLM-5-FP8 by up to 3.3x. The capability was initially validated on AWS Trainium clusters and contributed to vLLM as an accelerator-agnostic fe
Deepgram and AWS collaborated to port the Nova-3 speech-to-text model to AWS Trainium2. According to information published by AWS Machine Learning on October 20, 2026, a single trn2.3xlarge instance can run 8 model replicas, processing 11%–70% more audio per dollar in offline batch processing scenarios than the previous solution. The team used TorchNeuron to complete the PyTorch port and tuned it through NKILib fused kernels and hand-written NKI kernels, among which the FFN kernel latency was reduced by 2.0x, depthwise conv1d was sped up by 1.5x, and the conv2d kernel unlocked batch sizes above 32. There are currently no subsequent development facts.
Deepgram and AWS collaborated to port the Nova-3 speech-to-text model to AWS Trainium2. According to information published by AWS Machine Learning on October 20, 2026, a single trn2.3xlarge instance can run 8 model replicas, processing 11%–70% more audio per dollar in offline batch processing scenarios than the previous solution. The team used TorchNeuron to complete the PyTorch port and tuned it through NKILib fused kernels and hand-written NKI kernels, among which the FFN kernel latency was re
On October 20, 2026, AWS introduced a new bidirectional KV cache transfer mechanism in vLLM: in subsequent turns, the decode node sends the already-computed KV cache back to the prefill node, avoiding recomputation of the conversation history. On a 1P1D deployment on Amazon EC2 P5en, this mechanism improves TTFT for Qwen3-32B by up to 1.6x and for zai-org/GLM-5-FP8 by up to 3.3x. The capability was initially validated on AWS Trainium clusters and contributed to vLLM as an accelerator-agnostic feature that requires coordination with a KV-cache-aware router for cross-turn reuse. There are currently no further developments.
Deepgram and AWS collaborated to port the Nova-3 speech-to-text model to AWS Trainium2. According to information published by AWS Machine Learning on October 20, 2026, a single trn2.3xlarge instance can run 8 model replicas, processing 11%–70% more audio per dollar in offline batch processing scenarios than the previous solution. The team used TorchNeuron to complete the PyTorch port and tuned it through NKILib fused kernels and hand-written NKI kernels, among which the FFN kernel latency was reduced by 2.0x, depthwise conv1d was sped up by 1.5x, and the conv2d kernel unlocked batch sizes above 32. There are currently no subsequent development facts.