PR-OPD: Privileged Representation On-policy Self-Distillation for Agentic Reinforcement Learning
PR-OPD trains the agent to match hidden-state representations from its own skill-conditioned hindsight, not just token probabilities.
The paper says adding a skill prompt can improve WebShop success while changing fewer than a quarter of sampled token probabilities. The authors argue the stronger signal is inside the model’s hidden states, where skill effects show up across more than 80% of response tokens. Their method has the policy write a hindsight skill after a trajectory, reread its own response with that skill, and align hidden states layer by layer while still optimizing reward. In ALFWorld and WebShop, PR-OPD beats GRPO across the tested settings, with reported gains up to 4.7 success points and 14.0 accuracy points. HF Daily Papers' note
The paper says adding a skill prompt can improve WebShop success while changing fewer than a quarter of sampled token probabilities. The authors argue the stronger signal is inside the model’s hidden states, where skill effects show up across more than 80% of response tokens. Their method has the policy write a hindsight skill after a trajectory, reread its own response with that skill, and align hidden states layer by layer while still optimizing reward. In ALFWorld and WebShop, PR-OPD beats GRPO across the tested settings, with reported gains up to 4.7 success points and 14.0 accuracy points. HF Daily Papers' note
score 5