Desktop-Delta Bench: Do Computer-Use Models Understand Desktop GUI Transitions?
The benchmark tests whether desktop agents can tell what actually changed after an action.
Desktop-Delta Bench uses 2,013 human-verified Linux GUI steps across roughly 15 apps and 50 task domains. The paper says current tests miss this transition layer, where delayed screenshots, occlusions, or unrelated changes can be mistaken for progress. Across evaluated model families, temporal ordering remains far from solved, with the best exact-match rates around 65%. The authors frame DDB as a diagnostic benchmark for verification, reliability, and recovery in computer-use agents. ArXiv · AI/CL/LG's note
Desktop-Delta Bench uses 2,013 human-verified Linux GUI steps across roughly 15 apps and 50 task domains. The paper says current tests miss this transition layer, where delayed screenshots, occlusions, or unrelated changes can be mistaken for progress. Across evaluated model families, temporal ordering remains far from solved, with the best exact-match rates around 65%. The authors frame DDB as a diagnostic benchmark for verification, reliability, and recovery in computer-use agents. ArXiv · AI/CL/LG's note
score 5