A Theory of Policy Optimization in Post-Training: Coverage Collapse and Reference Resets
Rising reward can hide a collapse in the probability of generating optimal responses.
The authors show that REINFORCE, the basis for PPO and GRPO, can suffer this failure even with exact gradients and KL regularization toward the base model. Convergence can then require exponentially many steps despite good initial coverage of optimal responses. Their proposed framework repeatedly resets the KL reference to the latest policy, allowing stronger regularization while retaining convergence to global optimality. For softmax linear policies over pretrained features, they prove coverage-preserving guarantees for stylized PPO and GRPO variants and certain squared-loss algorithms.
ArXiv · AI/CL/LG's note
The authors show that REINFORCE, the basis for PPO and GRPO, can suffer this failure even with exact gradients and KL regularization toward the base model. Convergence can then require exponentially many steps despite good initial coverage of optimal responses. Their proposed framework repeatedly resets the KL reference to the latest policy, allowing stronger regularization while retaining convergence to global optimality. For softmax linear policies over pretrained features, they prove coverage-preserving guarantees for stylized PPO and GRPO variants and certain squared-loss algorithms.
ArXiv · AI/CL/LG's note
score 5