Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening
The paper says PPO critics can miss sharp value changes by smoothing them into nearly flat predictions.
The authors call the failure mode “Value Flattening,” seen in LLM reinforcement learning and in a controlled FrozenLake setup. They link it to an implicit variance penalty in critic loss and repeated updates from temporally correlated states. Their proposed fix, SP³O, supervises value loss on only a few well-separated states per response. On Qwen3-Base, using three supervised states per response reduced the effect and improved policy performance across tested model sizes and evaluation suites. HF Daily Papers' note
The authors call the failure mode “Value Flattening,” seen in LLM reinforcement learning and in a controlled FrozenLake setup. They link it to an implicit variance penalty in critic loss and repeated updates from temporally correlated states. Their proposed fix, SP³O, supervises value loss on only a few well-separated states per response. On Qwen3-Base, using three supervised states per response reduced the effect and improved policy performance across tested model sizes and evaluation suites. HF Daily Papers' note
score 5