β-OPSD: Deriving with Policy Optimization, Training with Self-Distillation
The paper reframes OPSD as a policy-optimization method with a tunable KL regularization weight, β.
Vanilla OPSD falls out as the β=1 case, which the authors argue helps explain why it can be brittle. Their β-OPSD target is a geometric interpolation between a reference policy and a privileged teacher, implemented by mixing token-level logits instead of running costly RL optimization. Return-to-go credit assignment is added to better match token updates to the sequence-level objective. On math reasoning benchmarks, the authors report better stability and downstream performance than vanilla OPSD. HF Daily Papers' note
Vanilla OPSD falls out as the β=1 case, which the authors argue helps explain why it can be brittle. Their β-OPSD target is a geometric interpolation between a reference policy and a privileged teacher, implemented by mixing token-level logits instead of running costly RL optimization. Return-to-go credit assignment is added to better match token updates to the sequence-level objective. On math reasoning benchmarks, the authors report better stability and downstream performance than vanilla OPSD. HF Daily Papers' note
score 5