More Methods, Better Agents? A Reality Check on Self-Distillation in Agentic Reinforcement Learning
None of the ten self-distillation methods consistently beat GRPO across the tested settings.
The researchers compared them on ALFWorld, ScienceWorld, Search-QA and WebShop across three model sizes. The study covered 132 training runs and 12,124.4 recorded H800 GPU-hours, with repeated evaluations over eight rollout seeds. Strong final scores sometimes masked unstable training, while extra training cost did not consistently improve performance. The benchmark addresses gaps in earlier evaluations, including baseline scores reused under differing experimental conditions.
HF Daily Papers' note
The researchers compared them on ALFWorld, ScienceWorld, Search-QA and WebShop across three model sizes. The study covered 132 training runs and 12,124.4 recorded H800 GPU-hours, with repeated evaluations over eight rollout seeds. Strong final scores sometimes masked unstable training, while extra training cost did not consistently improve performance. The benchmark addresses gaps in earlier evaluations, including baseline scores reused under differing experimental conditions.
HF Daily Papers' note
score 5