Regularized Score Centering
RSC with capped importance weights led all six agentic tasks in the paper’s final evaluations.
Asynchronous training can destabilize learning when data comes from stale policies. RSC adds a reverse-KL penalty to keep the learner’s next-token distribution close to the policy that generated the data. Combined with truncated importance sampling, it reached higher training rewards with the same number of sampled responses across all agentic benchmarks. The authors report a 14.3% improvement on AppWorld and 9.9 times the strongest baseline’s score on BabyAI.
HF Daily Papers' note
Asynchronous training can destabilize learning when data comes from stale policies. RSC adds a reverse-KL penalty to keep the learner’s next-token distribution close to the policy that generated the data. Combined with truncated importance sampling, it reached higher training rewards with the same number of sampled responses across all agentic benchmarks. The authors report a 14.3% improvement on AppWorld and 9.9 times the strongest baseline’s score on BabyAI.
HF Daily Papers' note
score 5