Right Screen, Wrong Transition: World Models as Verifiers for GUI Agents
LGWM detected hijacked GUI transitions with 0.987 AUC at 17 milliseconds per decision on the authors’ diagnostic benchmark.
A login screen can be legitimate after “Sign in” but an attack after “View order,” so the model checks whether the screen fits the action that preceded it. Trained on 1.85 million real GUI transitions, LGWM predicts the next screen’s encoded representation and compares it directly with the observed screen. On RSWT-BENCH, it matched the strongest closed-source vision-language models and outperformed generative GUI world models at over 1,000 times lower latency. Its mismatch signal also separated harmful from benign violations with 0.953 AUC.
HF Daily Papers' note
A login screen can be legitimate after “Sign in” but an attack after “View order,” so the model checks whether the screen fits the action that preceded it. Trained on 1.85 million real GUI transitions, LGWM predicts the next screen’s encoded representation and compares it directly with the observed screen. On RSWT-BENCH, it matched the strongest closed-source vision-language models and outperformed generative GUI world models at over 1,000 times lower latency. Its mismatch signal also separated harmful from benign violations with 0.953 AUC.
HF Daily Papers' note
score 5