Score Centering Stabilizes Off-policy Reinforcement Learning
The paper pins RL instability on drift between rollout and training engines, then cancels it with an additive correction.
Marek and Ryabinin argue that training-inference mismatch does not need to be eliminated outright. Their “score centering” term is designed to remove the persistent bias that compounds across updates. In tests from 0.6B to 30B parameters, it matched or beat importance-sampling methods under quantization, with bigger gains under worse mismatch. The authors also report that it composes with importance sampling in staleness experiments.
ArXiv · AI/CL/LG's note
Marek and Ryabinin argue that training-inference mismatch does not need to be eliminated outright. Their “score centering” term is designed to remove the persistent bias that compounds across updates. In tests from 0.6B to 30B parameters, it matched or beat importance-sampling methods under quantization, with bigger gains under worse mismatch. The authors also report that it composes with importance sampling in staleness experiments.
ArXiv · AI/CL/LG's note
score 6