Le Critique: Privileged Value Functions for LLM Reinforcement Learning
The paper argues that critics can earn their keep in LLM RL if they get privileged token-level signal and a fallback when they are wrong.
Venkatraman, Dinot, and Aitchison propose Privileged Value Functions, meant to add task-relevant token-level information without biasing the policy objective. They also introduce TETHER, which shifts between group-relative and value baselines based on how accurate the value function is. In reasoning-task experiments, the two methods improve on a standard value-function baseline and match or beat mean-baseline GRPO. ArXiv · AI/CL/LG's note
Venkatraman, Dinot, and Aitchison propose Privileged Value Functions, meant to add task-relevant token-level information without biasing the policy objective. They also introduce TETHER, which shifts between group-relative and value baselines based on how accurate the value function is. In reasoning-task experiments, the two methods improve on a standard value-function baseline and match or beat mean-baseline GRPO. ArXiv · AI/CL/LG's note
score 5