Coding-Agent Benchmarks Should Match Their Users' Task Flows
Real JetBrains coding-agent sessions did not look like today’s issue-based benchmarks.
The paper analyzes 4,782 production agent sessions and focuses on longer interactive sessions with at least three user messages. Those sessions mixed code questions, planning, review, refactoring, execution, and other task types, often shifting mid-session. The authors propose SWE-TaskFlow to reshape issue-derived benchmarks toward a measured target interaction pattern while keeping verified tasks and tests. In a 700-task SWE-Bench Pro pilot, splitting work into sequential steps roughly doubled agent cost without a stable resolve-rate change. HF Daily Papers' note
The paper analyzes 4,782 production agent sessions and focuses on longer interactive sessions with at least three user messages. Those sessions mixed code questions, planning, review, refactoring, execution, and other task types, often shifting mid-session. The authors propose SWE-TaskFlow to reshape issue-derived benchmarks toward a measured target interaction pattern while keeping verified tasks and tests. In a 700-task SWE-Bench Pro pilot, splitting work into sequential steps roughly doubled agent cost without a stable resolve-rate change. HF Daily Papers' note
score 5