Inducing Task Models from Computer-Use Traces
The paper proposes TMI, a method for turning raw computer-use recordings into auditable task models.
It targets traces made of screenshots plus mouse and keyboard actions, where multiple goals may be interleaved. TMI first separates latent tasks, then builds both a hierarchical objective model and a control-flow procedure model for each one. In tests, it reports 0.974 agreement with ground-truth task groupings, reconstructs 74.9% of observed execution steps, and improves held-out task accuracy by 30.0% over the strongest baseline. ArXiv · AI/CL/LG's note
It targets traces made of screenshots plus mouse and keyboard actions, where multiple goals may be interleaved. TMI first separates latent tasks, then builds both a hierarchical objective model and a control-flow procedure model for each one. In tests, it reports 0.974 agreement with ground-truth task groupings, reconstructs 74.9% of observed execution steps, and improves held-out task accuracy by 30.0% over the strongest baseline. ArXiv · AI/CL/LG's note
score 5