Tencent research extends critical batch size theory to online LLM reinforcement learning

Tencent has released new research that applies classical critical-batch-size theory to online reinforcement learning for large language models. In online RL, the model generates its own training data, and rollout generation and training scale differently.

Experiments across GRPO and PPO showed that retuning the learning rate can preserve learning per response over a bounded range of batch sizes. On fixed hardware, increasing batch size improved PPO generation-stage throughput by up to 2.29×, while the best measured GRPO configuration reached the same validation target in 29% less time.