Two Clocks: Why Larger Rollout Batches Learn Slower but Fail on Schedule in Reinforcement Learning with Verifiable Rewards
The paper argues that bigger RLVR rollout batches can spend the same failure budget while doing less useful learning.
The authors separate training into a progress clock and a failure clock: samples needed to improve versus samples consumed before collapse. Their local model says square-root learning-rate scaling keeps drift roughly aligned per sample, but larger batches make fewer, longer updates and lose efficiency to curvature. GRPO experiments on Qwen2.5-Math-1.5B support that split across a 16x batch range, with ratio clipping delaying failure but not closing the progress gap. They propose starting below the critical batch size, then switching larger before failure. ArXiv · AI/CL/LG's note
The authors separate training into a progress clock and a failure clock: samples needed to improve versus samples consumed before collapse. Their local model says square-root learning-rate scaling keeps drift roughly aligned per sample, but larger batches make fewer, longer updates and lose efficiency to curvature. GRPO experiments on Qwen2.5-Math-1.5B support that split across a 16x batch range, with ratio clipping delaying failure but not closing the progress gap. They propose starting below the critical batch size, then switching larger before failure. ArXiv · AI/CL/LG's note
score 5