Predictive Divergence Masks for LLM RL

The paper targets a mismatch inside PPO-style masking: the ratio test can point the wrong way against the divergence it is meant to control.

The authors propose a predictive divergence mask that estimates whether the next policy-gradient step will raise or lower the trust-region divergence. For LLM softmax policies, they derive the prediction in closed form, then add two top-K estimators for rollout systems that only expose truncated vocabularies. They report better alignment with actual divergence changes and improved RL training across model scales and precision settings. HF Daily Papers' note