TPPO: Tracking-Value PPO For Language Model Reasoning
TPPO changes PPO’s critic so it tracks token-level value shifts under sparse final rewards.
The paper models successive prefix values as additive updates in log-odds space, then sums position-wise scalar outputs into value logits. Those reconstructed values are trained from terminal outcomes. The authors report math-reasoning gains over PPO of 7.0 points on Qwen3-4B-Base and 8.4 points on Qwen3-8B-Base, with improvements also shown on other Qwen models, countdown, and scientific coding tasks. HF Daily Papers' note
The paper models successive prefix values as additive updates in log-odds space, then sums position-wise scalar outputs into value logits. Those reconstructed values are trained from terminal outcomes. The authors report math-reasoning gains over PPO of 7.0 points on Qwen3-4B-Base and 8.4 points on Qwen3-8B-Base, with improvements also shown on other Qwen models, countdown, and scientific coding tasks. HF Daily Papers' note
score 5