StateAct: Program State, before Pixels, for Long-Horizon Computer-Use Agents
StateAct reports better OSWorld 2.0 results by letting the agent work on program state directly, using pixels only when needed.
The harness gives the main agent code access to files, app backends, DOM state, and other task data, with a GUI subagent reserved for screenshot-and-click subgoals. In the paper’s run, that GUI path was used on 28 of 108 tasks and 1.1% of main-agent steps. An independent finish gate checks whether outputs are missing, unsaved, or saved to the wrong place. On OSWorld 2.0, it raises Claude Opus 4.8 from 20.6% to 26.9% binary success, while cutting cost per task by about 9x versus a screenshot-only setup. HF Daily Papers' note
The harness gives the main agent code access to files, app backends, DOM state, and other task data, with a GUI subagent reserved for screenshot-and-click subgoals. In the paper’s run, that GUI path was used on 28 of 108 tasks and 1.1% of main-agent steps. An independent finish gate checks whether outputs are missing, unsaved, or saved to the wrong place. On OSWorld 2.0, it raises Claude Opus 4.8 from 20.6% to 26.9% binary success, while cutting cost per task by about 9x versus a screenshot-only setup. HF Daily Papers' note
score 6