Parameter Exploration for RLVR via Variational Learning
The paper tests parameter-space exploration as a way to improve RL training for LLMs without adding meaningful compute cost.
The authors introduce Perturbed Parameter Policy Optimization, or 3PO, which samples different policies from a posterior to generate rollouts. In experiments on OLMo-3-1025-7B and Qwen2.5-Math-7B, the methods beat standard GRPO on average across math reasoning and code generation tasks at near-identical FLOPs cost. Multiple parameter samples also produced fewer zero-advantage groups and fewer malformed or incorrect rollouts than GRPO and action-space baselines. Source: HF Daily Papers' note.
The authors introduce Perturbed Parameter Policy Optimization, or 3PO, which samples different policies from a posterior to generate rollouts. In experiments on OLMo-3-1025-7B and Qwen2.5-Math-7B, the methods beat standard GRPO on average across math reasoning and code generation tasks at near-identical FLOPs cost. Multiple parameter samples also produced fewer zero-advantage groups and fewer malformed or incorrect rollouts than GRPO and action-space baselines. Source: HF Daily Papers' note.
score 5