Google Releases TPU 8i Max and TPU 8tx, Doubling HBM Bandwidth

Chip Inside, November 9 — Just now, Google released two TPUs: the TPU 8i Max, designed for ultra-large context windows and focused on the highest compute density per rack, and the TPU 8tx, which supports secure and confidential inference.

Google says that compared with the TPU 8t, the TPU 8tx can cut latency by up to half with almost negligible performance loss; the TPU 8i Max's compute, SRAM, and HBM bandwidth are all twice those of the TPU 8i.

This is the first time Google has launched four TPUs within a single year. In April this year, Google launched two new specialized TPUs: the 8t for large-scale training and the 8i for real-time inference.

With this, the four TPUs in Google's eighth-generation TPU lineup can cover the entire agent lifecycle.

In addition to launching the latest TPUs, Google also upgraded a unified open-source software stack for developers.

1. Targeting confidential computing and the memory wall, Google rolls out 2 TPUs

Google had previously been collaborating with Google DeepMind to design TPUs, but as the computing paradigm shifts from human interaction to agent-oriented tasks, enterprises need accelerators capable of completing inference within ultra-large context windows. Some customers in heavily regulated industries also require dedicated encryption capabilities for their specific business workloads.

That is why Google launched two new TPUs this time, with the goal of making TPUs easily accessible to every developer and cloud customer.

Google's eighth-generation TPU portfolio targets different stages of the AI lifecycle:

The TPU 8t released in April this year was designed as the training core for large foundation models, aiming for the highest "effective throughput," that is, the effective intelligence output per watt of electricity; the TPU 8i is an inference engine designed to break the memory wall, cutting inter-chip communication latency by up to half.

The newly launched TPU 8tx and TPU 8i Max target confidential computing and the memory wall.

The TPU 8tx is the third TPU in Google's eighth generation, optimized for secure and confidential inference and also suitable for small-scale training. It is built for regulated enterprises and government agencies, which can run frontier models and agents while processing sensitive data.

Normally, data is encrypted when stored or transmitted over networks, but during computer processing it must be decrypted in memory. The TPU 8tx uses the same chip as the TPU 8t, paired with a fifth-generation AMD EPYC CPU host, building security capabilities directly into the hardware layer so that even when sensitive data and model weights are in active use during computation, they remain continuously protected and isolated.

At the same time, the TPU 8tx uses Google's low-latency Boardfly network built for real-time inference, ensuring smooth operation of multi-step agent tasks. Google claims that compared with the TPU 8t, the TPU 8tx can cut latency by up to half with almost negligible performance loss.

Combining training-grade compute with a high-speed inference network also means enterprises can run and customize models on the same system at the same time. With the TPU 8tx, R&D teams can fine-tune models with private data, distill large models into small models, and train agents through reinforcement learning, all without migrating sensitive data between multiple separate clusters.

The TPU 8i Max addresses the memory wall problem.

When dozens of agents work together across an entire software codebase or hours of video content at the same time, memory bandwidth bottlenecks become prominent. So Google packaged two low-latency TPU 8i chips into the same package substrate.

Google says the TPU 8i Max can achieve synchronized scaling of compute and memory, with compute, SRAM, and HBM bandwidth all twice those of the TPU 8i.

In addition, the TPU 8i Max integrates twice the chip area within the original rack space, so data centers can deploy more compute with the same footprint and the same power consumption.

2. Google wants to open TPUs to all developers, with a unified software stack upgrade

Google also announced that, in order to open TPUs to all developers, it will invest in the following areas:

First, Google will open TPU development capabilities to all PyTorch developers without requiring them to rewrite code.

Last month, Google open-sourced TorchTPU, a native PyTorch framework for TPUs. Now PyTorch code and models on Hugging Face can run natively on TPUs, while mainstream open-source inference engines such as vLLM and SGLang also provide out-of-the-box TPU backends.

In addition, to lower the barrier to hardware optimization, Google's open-source TPU agent toolkit can plug directly into programming tools such as Claude Code, Cursor, and Google Antigravity, completing TPU environment onboarding in minutes, automatically locating performance bottlenecks, and supporting automatic tuning of custom operators using natural language. The toolkit is currently still in preview.

For production deployment scenarios, Google has open-sourced the XLA Serving Engine workflow, which supports model training in Python and then direct deployment in a low-latency C++ runtime, providing services across heterogeneous hardware including TPUs, GPUs, and CPUs.

Second, Google says its TPUs have achieved industry-leading production-grade performance on the InferenceX and AgentX benchmarks.

The industry-standard benchmark suites InferenceX and AgentX target real-world agent code generation and long-context tasks. According to test results published by Google, relying on the native vLLM-TorchTPU software stack on TPUs, it achieved current top-tier cost-performance. This advantage can cover multiple frontier open-source models as well as various open-source serving stacks, while ensuring model accuracy is unaffected.

Third, Google will further eliminate the wasted compute problem of autonomous agents.

AI agents are stateful programs that often pause for several seconds or even hours while waiting for database query results, external tool responses, or human feedback. To make accelerators more power-efficient during such waiting periods, Google's Agent Compute Engine software can automatically suspend idle agent sessions and migrate working memory off the chip within milliseconds.

Fourth, Google will break through the memory wall with TPU Sync.

Google will open-source TPU Sync, a model state layer for TPU clusters. TPU Sync fuses chip HBM, host DRAM, shared storage, and the data center network into a co-designed unified memory hierarchy; it can automatically offload KV cache to outside the chip within milliseconds, stream context between disaggregated prefill nodes and decode nodes, synchronize reinforcement learning weights, and ultimately reduce KV transfer latency and time to first token by more than 90%.

Fifth, Google will horizontally scale distributed tasks across multiple TPUs through a single controller.

Google's self-developed large-scale distributed machine learning runtime system, Cloud Pathways, is now generally available on Google Kubernetes Engine (GKE), opening the Google DeepMind runtime to all Google Cloud customers.

Relying on a single-controller architecture and Pathways tools, developers can orchestrate large-scale TPU clusters spanning multiple TPU slices. Google says that whether for pre-training, reinforcement learning, or multi-host inference, developers can use it as if operating a single local machine.

In addition, with co-located Python sidecars, sub-minute TPU slice hot-swapping, and elastic shard resharding capabilities, Cloud Pathways enables long-running jobs to keep progressing when hardware failures occur.

Finally, Google will leverage GKE dynamic slicing capabilities to assemble custom TPU topologies on demand.

GKE dynamic slicing is now generally available. Using optical circuit switching (OCS) technology, it decouples physical hardware resource provisioning from logical slice formation. This feature can dynamically combine TPU cubes into larger topologies (super-slices) in real time, or split them into smaller, isolated workloads (sub-slices).

Google says this topological flexibility can shorten job startup time to as little as one-fifth of the original and improve failure recovery speed by up to 4.5 times, maximizing resource utilization and effective job throughput of the entire TPU pool.

Conclusion: In the agent era, are accelerators heading toward scenario-specific customization?

Google says that today more and more enterprises are using AI to conceive and build next-generation AI infrastructure, and Google is doing the same. Whether using Google DeepMind's AlphaChip reinforcement learning system or relying on Gemini to complete chip layout design within hours, Google is using TPU-based AI services to accelerate next-generation TPU hardware development.

Now, with the explosive growth of compute demand for agent scenarios, compute vendors are shifting from general-purpose accelerators to segmented optimization for scenarios such as training, inference, confidentiality, and long context. Google has formed a closed loop through chips, models, and systems reinforcing one another, creating this accelerator solution customized around the agent lifecycle—a new exploration aimed at the above trends.

Source: Google