$β$-OPSD: Deriving with Policy Optimization, Training with Self-Distillation
The paper recasts OPSD as a tunable policy-optimization objective, with β controlling how tightly the student stays anchored to a reference policy.
The authors say vanilla OPSD is the β=1 case of a broader family, making its regularization strength explicit instead of fixed. Their β-OPSD target is a geometric interpolation between the reference policy and a privileged teacher, implemented by mixing token-level logits. They avoid direct reinforcement learning by distilling toward the closed-form policy target, with return-to-go credit assignment tying token updates back to sequence-level rewards. On mathematical reasoning benchmarks, they report better stability and downstream performance than vanilla OPSD. ArXiv · AI/CL/LG's note
The authors say vanilla OPSD is the β=1 case of a broader family, making its regularization strength explicit instead of fixed. Their β-OPSD target is a geometric interpolation between the reference policy and a privileged teacher, implemented by mixing token-level logits. They avoid direct reinforcement learning by distilling toward the closed-form policy target, with return-to-go credit assignment tying token updates back to sequence-level rewards. On mathematical reasoning benchmarks, they report better stability and downstream performance than vanilla OPSD. ArXiv · AI/CL/LG's note
score 4