Google Exec: AI Industry Bottleneck Shifts from Compute to Memory, Which Now Accounts for ~75% of AI Server BOM

Google's senior director of supply chain infrastructure, Nikhil Cherian, recently said that the AI industry's bottleneck has shifted from compute to memory, with high-performance memory now accounting for about 75% of the material cost of a single AI server.

To address the challenge, Google is working on both hardware and software. On the hardware side, Google has set up an internal recycling supply chain that harvests usable DDR4 modules from decommissioned servers, and has designed hardware adapters so the older DDR4 memory can be connected to next-generation AI servers. The company is also re-importing retired servers specifically to extract their DDR4 modules.

On the software side, Google continues to optimize low-level libraries, model architectures, and KV cache compression techniques to reduce memory consumption per unit of compute.

Earlier this year, Google introduced two TPU ASICs: TPU8t for training and TPU8i for inference. The TPU8i packs 288GB of HBM3e per chip, provides 8.6 TB/s of memory bandwidth, and integrates 384MB of on-chip SRAM to store active KV caches, reducing calls to external system memory. TPU8i servers use Google's Arm-based Axion processors instead of traditional x86 host CPUs, relying on a high-speed DDR5 memory architecture for host-side tasks such as data preprocessing.