Bellman Policy Optimization
BPO reframes policy mirror descent for terminal-reward LLM training without estimating intermediate state values.
The paper introduces Bellman Policy Optimization, a critic-free RLVR method for autoregressive generation. It uses Bellman equations to turn the PMD objective into a trajectory-level objective, then approximates that into a practical loss. The authors say the reformulated objective has the same unique optimum as the original PMD objective. Experiments on mathematical reasoning benchmarks are reported as showing BPO’s effectiveness. HF Daily Papers' note
The paper introduces Bellman Policy Optimization, a critic-free RLVR method for autoregressive generation. It uses Bellman equations to turn the PMD objective into a trajectory-level objective, then approximates that into a practical loss. The authors say the reformulated objective has the same unique optimum as the original PMD objective. Experiments on mathematical reasoning benchmarks are reported as showing BPO’s effectiveness. HF Daily Papers' note
score 6