Baselines and Score Centering in Policy Gradient Estimation
With regularized token-level importance sampling, a group-mean baseline generally introduces bias unless score centering is also used.
The note examines off-policy training, where responses come from a different policy than the one being updated. Truncating, masking or clipping importance ratios makes the gradient estimate sensitive to constant shifts in reward; a covariance formulation restores that invariance. The group-mean baseline and score centering subtract different means, producing distinct unbiased covariance estimators with different variances. For sequence-level importance sampling, score centering adds no benefit once the group-mean baseline is used.
ArXiv · AI/CL/LG's note
The note examines off-policy training, where responses come from a different policy than the one being updated. Truncating, masking or clipping importance ratios makes the gradient estimate sensitive to constant shifts in reward; a covariance formulation restores that invariance. The group-mean baseline and score centering subtract different means, producing distinct unbiased covariance estimators with different variances. For sequence-level importance sampling, score centering adds no benefit once the group-mean baseline is used.
ArXiv · AI/CL/LG's note
score 5