WEKA field CTO: GPUs can spend up to 60% of compute recomputing LLM context

At the VentureBeat AI Impact Series event in October 2026, WEKA field CTO Alan McSeveney told lead analyst Rob Strechay that GPUs running large language model inference can spend up to 60% of compute recalculating context they have already processed. The problem, he explained, stems from the way LLMs keep conversation history in GPU memory as a KV cache; when older cache entries are evicted due to memory limits, the GPU may have to reconstruct the same history when a session resumes.

Without a cache-management strategy, a GPU can sit as low as 30% utilization on new work, McSeveney said. With a good strategy, utilization can rise above 90%, reaching 95% in the best case. Since GPUs can account for more than 80% of the budget in an AI infrastructure build, spending cycles on recomputation is a colossal waste of money. He noted that organizations are constrained by budget, power availability and data center space, driving them to get more value from existing hardware.

McSeveney pointed to industry moves toward more aggressive KV-cache reuse and offload, including Mooncake, recent DeepSeek techniques, and Nvidia's CMX context memory platform in the Vera Rubin stack. He argued that NVMe flash is a more economical place to hold the cache than expanding server DRAM, matching or beating DRAM for KV-cache storage while scaling better and costing less. WEKA's Augmented Memory Grid, built on its NeuralMesh storage software, captures KV cache from GPU servers, stores it centrally, and loads it back into any host in the data center, allowing sessions to move between servers without recomputation.

Compared with offloading to DRAM, central KV-cache loading increased the number of users on the same GPUs by 7x and lifted output from 200,000 to 2 million tokens in the same timeframe, McSeveney said. He added that optimizing the cache does not remove the need to optimize the application, workflow and pipeline.