SPO++: Stream-Aligned Policy Optimization for Asynchronous Agentic RL
SPO++ targets a mismatch in SPO’s advantage normalization for asynchronous tool-use reinforcement learning.
The paper says SPO avoids waiting for sibling rollouts, but its per-trajectory whitening does not align with the token-weighted actor loss. SPO++ instead standardizes terminal-outcome advantages under the action-token measure and groups prompt evidence by the policy event that produced it. In matched ALFWorld and Math-TIR runs, the authors report better online learning efficiency than SPO. Their paired ablation names action-token-measure normalization as the strongest tested component. ArXiv · AI/CL/LG's note
The paper says SPO avoids waiting for sibling rollouts, but its per-trajectory whitening does not align with the token-weighted actor loss. SPO++ instead standardizes terminal-outcome advantages under the action-token measure and groups prompt evidence by the policy event that produced it. In matched ALFWorld and Math-TIR runs, the authors report better online learning efficiency than SPO. Their paired ablation names action-token-measure normalization as the strongest tested component. ArXiv · AI/CL/LG's note
score 5