Information-Time Proximal Policy Optimization
InfoPPO treats reasoning steps by information density, not token count.
The paper argues that token-by-token time makes PPO discounting poorly matched to LLM reasoning, where information arrives unevenly. Its method ties credit propagation and policy-update limits to state-wise information density, including adaptive clipping at each token position. In experiments on Qwen3 models, the authors report gains across five competition-style math benchmarks and steadier accuracy and response length under non-trivial discounting. ArXiv · AI/CL/LG's note
The paper argues that token-by-token time makes PPO discounting poorly matched to LLM reasoning, where information arrives unevenly. Its method ties credit propagation and policy-update limits to state-wise information density, including adaptive clipping at each token position. In experiments on Qwen3 models, the authors report gains across five competition-style math benchmarks and steadier accuracy and response length under non-trivial discounting. ArXiv · AI/CL/LG's note
score 5