The Next Screenshot Knows: Gated Hindsight Distillation for Mobile GUI Agents
The paper trains mobile GUI agents with “hindsight” from the next screen, but only when that extra view fixes a failed action.
Standard offline training drops the subsequent observation, even when that screen explains why the previous tap was right. Gated Hindsight Distillation adds a teacher model that can see the next screenshot and re-score the student’s attempted responses. The distillation is gated: it is used only when the student misses and the hindsight-conditioned teacher recovers the demonstrated action. The authors report better task success than GRPO on AndroidWorld and AndroidLab across two vision-language models. HF Daily Papers' note
Standard offline training drops the subsequent observation, even when that screen explains why the previous tap was right. Gated Hindsight Distillation adds a teacher model that can see the next screenshot and re-score the student’s attempted responses. The distillation is gated: it is used only when the student misses and the hindsight-conditioned teacher recovers the demonstrated action. The authors report better task success than GRPO on AndroidWorld and AndroidLab across two vision-language models. HF Daily Papers' note
score 5