NVIDIA's vLLM Optimization Boosts Kimi K3 Inference Speed by 1.68x

On October 18, research firm SemiAnalysis reported that NVIDIA has delivered an upstream vLLM optimization that enables the 2.8-trillion-parameter Kimi K3 model to reach 423 tokens per second per user on GB300 NVL72 hardware, a 1.68x improvement over previous performance.

The optimization was carried out by Benjamin, an engineer at NVIDIA's CentML team, using large-scale megakernel fusion and a decoding-specific kernel written with the Cute DSL. The megakernel comprises approximately 12,000 lines of code and is seen as a direction for AI-assisted generation of advanced compute kernels.

NVIDIA is also developing vLLM optimizations for other models, including DeepSeek V4.1.