SemiAnalysis Report Breaks Down LLM Inference as a Four-Stage 'Token Factory'

On September 22, research firm SemiAnalysis published an in-depth technical report that deconstructs the underlying architecture of large language model (LLM) inference services. The report breaks the inference process into four working stages—Prefill, Midfill, Decode Attention, and Decode Experts—together described as a “token factory.”

According to the report, these stages place very different demands on compute, memory bandwidth, and network resources. Prefill is compute-bound as it processes many new tokens at once, while Midfill appends a small number of tokens to an existing long cache prefix, combining large KV-state reads with weight reuse. Decode stages are largely dominated by memory bandwidth, and attention reads can even exceed model weight size in long-context scenarios. Treating these stages as a single workload, the report argues, undermines the structural advantages of mixture-of-experts (MoE) architectures, where a router activates only a small subset of expert modules per token.

Using Nvidia’s Blackwell-series GPUs as a case study, the report also presents performance projections from SemiAnalysis’ inference simulator for the Kimi K3 model across B200, B300, and GB200 systems. GB200 leads on much of the latency-throughput trade-off curve in decode, while peak HBM residency per GPU stays below 80GB in most frontier configurations. The authors contend that memory bandwidth often generates more revenue than raw capacity in inference, and expensive HBM should be reserved for active batch data, with idle KV states migrated to CPU DRAM or cheaper network-attached storage.

The report further highlights the importance of scheduling, warns about “retirement-gated oscillation” in large clusters, and compares disaggregated versus aggregated architecture trade-offs. It provides a practical framework for hardware designers, cloud providers, and model deployers.