Interactive Visual Gym: Evaluating Visual Interaction Models with Simulated Users in Interactive Worlds
The best-performing model completed just 53.4% of tasks within the time limit in a benchmark where its advice changes what simulated users do next.
IVG-Bench contains 315 interaction scenarios across cooking, household manipulation and mobile interfaces, with simulated users who can make execution errors or stray from plans. The strongest model tested, Gemini 4 Argon, struggled with factual grounding and actionable guidance; accounting for response latency also showed it was often too slow to prevent errors. A human study found 78.5% agreement between the benchmark’s interaction-quality metric and participant preferences. A proof-of-concept training experiment improved Qwen3.5-4B’s task success and interaction quality in the cooking environment.
HF Daily Papers' note
IVG-Bench contains 315 interaction scenarios across cooking, household manipulation and mobile interfaces, with simulated users who can make execution errors or stray from plans. The strongest model tested, Gemini 4 Argon, struggled with factual grounding and actionable guidance; accounting for response latency also showed it was often too slow to prevent errors. A human study found 78.5% agreement between the benchmark’s interaction-quality metric and participant preferences. A proof-of-concept training experiment improved Qwen3.5-4B’s task success and interaction quality in the cooking environment.
HF Daily Papers' note
score 5