Bellman Policy Optimization
BPO trains RLVR language models without estimating intermediate state values.
The paper derives Bellman Policy Optimization from Policy Mirror Descent for autoregressive generation with terminal rewards. Its Bellman reformulation turns the objective into a trajectory-level loss while preserving the same unique optimum as the original PMD objective. The practical loss uses a mismatch-correction weight based on a smoothed ratio of complementary token probabilities. The authors report gains on mathematical reasoning benchmarks. ArXiv · AI/CL/LG's note
The paper derives Bellman Policy Optimization from Policy Mirror Descent for autoregressive generation with terminal rewards. Its Bellman reformulation turns the objective into a trajectory-level loss while preserving the same unique optimum as the original PMD objective. The practical loss uses a mismatch-correction weight based on a smoothed ratio of complementary token probabilities. The authors report gains on mathematical reasoning benchmarks. ArXiv · AI/CL/LG's note
score 5