Enabling Token-wise Importance Sampling in Off-Policy RLVR with Proximal KL Regularization
The paper argues Proximal KL can make token-wise off-policy RLVR updates work without full-response weighting or a fitted critic.
The authors prove conditions under which future KL credit preserves the intended proximal optimizer even when training on mixtures of older policies. They report matched off-policy tests on Qwen3-1.7B and Qwen3-4B where Proximal KL beats GRPO by up to 2.58% micro-averaged accuracy across eight math benchmarks. Periodic-refresh runs also show it avoiding collapses seen in PG and CISPO while reaching higher accuracy in fewer updates. Source: ArXiv · AI/CL/LG's note.
The authors prove conditions under which future KL credit preserves the intended proximal optimizer even when training on mixtures of older policies. They report matched off-policy tests on Qwen3-1.7B and Qwen3-4B where Proximal KL beats GRPO by up to 2.58% micro-averaged accuracy across eight math benchmarks. Periodic-refresh runs also show it avoiding collapses seen in PG and CISPO while reaching higher accuracy in fewer updates. Source: ArXiv · AI/CL/LG's note.
score 5