NVIDIA introduces the five co-design elements of Vera Rubin NVL72

Nov 6

1. 11:18

   IT Home · Artificial Intelligence

   NVIDIA Explains the AI Memory Bottleneck: HBM Grows Only About 3%, Agent Throughput per Dollar Improves by Up to 30x

   NVIDIA released a technical explainer proposing full-stack co-design to alleviate the AI memory bottleneck, co-designing GPU, CPU, network, storage, and software as a whole. The plan includes compressing model weights and KV cache using the low-precision NVFP4 format, distributing model weights across up to 72 GPUs in the NVL72 system, buffering paused context in the LPDDR memory of the Vera CPU, and migrating inactive context to CMX context storage via the Dynamo inference software. Citing the SemiAnalysis AgentX benchmark, NVIDIA claims that at 125 tokens per second per user, the Vera Rubin NVL72 achieves up to 30x higher agent throughput per dollar compared with the GB300 NVL72, while HBM capacity per GPU increases by only about 3%.

   Expand summary
2. 11:18

   IT Home · AIRelated discussion

   IT Home · AI joined this discussion ↗

   This post contributes to heat; it is not a formal report or confirmed fact.

3. 11:15

   Semiconductor Industry Observer

   NVIDIA's HBM capacity grows only 3%, while AI inference throughput per unit cost improves by up to 30x

   NVIDIA says that from the Blackwell Ultra to the Vera Rubin platform, HBM capacity per GPU grows by only about 3%, but according to SemiAnalysis AgentX testing, throughput per unit cost for agentic workloads improves by up to 30x. Its approach is not to stack more HBM, but to manage the memory hierarchy across GPU, CPU, SSD and shared storage through full-stack co-design, including compressing model weights and the KV Cache with NVFP4, offloading inactive context to CMX context memory storage, and compressing RAG vectors with the Nemotron Retriever. At a generation speed of 125 Token/s per user, Vera Rubin NVL72 achieves up to 30x higher throughput per unit cost than GB300 NVL72.

   Expand summary
4. 07:37

   Sina Tech

   NVIDIA Explains How to Break the AI Memory Bottleneck: HBM Grows Only About 3%, Agent Throughput per Dollar Up to 30x Higher

   NVIDIA released a technical explainer proposing full-stack co-design to alleviate the AI memory bottleneck: with HBM capacity increased by only about 3%, Vera Rubin NVL72 achieves up to 30x higher agent throughput per dollar compared with GB300 NVL72. Specific measures include using the NVFP4 low-precision format to compress model weights and KV cache, distributing model weights across up to 72 GPUs in the NVL72 system, buffering paused context in the LPDDR memory of the Vera CPU, and using the Dynamo inference software to migrate inactive context to CMX context storage. NVIDIA cited benchmark data from SemiAnalysis AgentX and noted that the key to compute competition is shifting from memory capacity to how many effective tokens can be produced per byte and per watt.

   Expand summary
5. 00:00

   NVIDIA Developer · Generative AIFirst-party

   How NVIDIA Co-Design Maximizes the Value of Every Byte in the AI Factory: Five Ways

   Through five co-designs—NVFP4, Wide-EP, Dynamo, Vera CPU, and CMX—NVIDIA enables Vera Rubin NVL72 to achieve up to 30x the per-megawatt throughput of GB300 NVL72 on SemiAnalysis AgentX while keeping 288 GB of HBM per GPU unchanged. NVFP4 reduces model footprint by about 3.5x versus FP16 and 1.8x versus FP8, and NVFP4 KV cache cuts it by up to another 50% versus FP8. TensorRT-LLM's Wide-EP spreads experts across 8 or more GPUs, while Dynamo manages KV cache as a cluster-level resource across GPUs, CPUs, and storage.

   Expand summary