Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agents
AlignOPSD tries to fix a credit problem in long-horizon agent training: the useful decision may not sit at the timestamp where supervision is applied.
The paper defines this as Decision-Timestamp Mismatch, where privileged feedback can be out of sync with the student agent’s functional decision or too narrow for a decision that spans several steps. Its method, AlignOPSD, first re-scores student responses in matched contexts across sibling rollouts, then assigns credit over variable-duration decision spans. Tested with Qwen2.5-3B and Qwen2.5-7B on ALFWorld, WebShop, and Search-QA, it beats GRPO and StepOPSD across all eight backbone-metric comparisons. The authors report a 5.5% to 8.7% gain over GRPO and first-place ranking in six comparisons. HF Daily Papers' note
The paper defines this as Decision-Timestamp Mismatch, where privileged feedback can be out of sync with the student agent’s functional decision or too narrow for a decision that spans several steps. Its method, AlignOPSD, first re-scores student responses in matched contexts across sibling rollouts, then assigns credit over variable-duration decision spans. Tested with Qwen2.5-3B and Qwen2.5-7B on ALFWorld, WebShop, and Search-QA, it beats GRPO and StepOPSD across all eight backbone-metric comparisons. The authors report a 5.5% to 8.7% gain over GRPO and first-place ranking in six comparisons. HF Daily Papers' note
score 5