Stale but Stable: Staleness-Adaptive Trust Regions for Stabilizing Asynchronous Reinforcement Learning
SAT tightens PPO updates only where stale asynchronous rollouts create the highest policy mismatch.
The paper frames staleness as a trust-region problem caused by policy lag, engine delays, and mixture-of-experts routing. Its Staleness-Adaptive Trust Region uses detached sampled log-ratios as a proxy, finds the high-mismatch tail in each batch, and contracts the relevant PPO clip endpoint for outward updates. In experiments on Qwen3-30B-A3B-Base with SGLang inference and Megatron training, SAT-GSPO w/ R3 reports the best AIME24 avg@8: 35.83 at lag 1 and 34.79 at lag 8. The authors argue the gains come from matching the clip interval to observed staleness heterogeneity rather than treating all tokens the same. HF Daily Papers' note
The paper frames staleness as a trust-region problem caused by policy lag, engine delays, and mixture-of-experts routing. Its Staleness-Adaptive Trust Region uses detached sampled log-ratios as a proxy, finds the high-mismatch tail in each batch, and contracts the relevant PPO clip endpoint for outward updates. In experiments on Qwen3-30B-A3B-Base with SGLang inference and Megatron training, SAT-GSPO w/ R3 reports the best AIME24 avg@8: 35.83 at lag 1 and 34.79 at lag 8. The authors argue the gains come from matching the clip interval to observed staleness heterogeneity rather than treating all tokens the same. HF Daily Papers' note
score 4