Efficient SWE Agent Benchmarking via Trajectory-Aware Evaluation
The paper’s claim is that SWE agent scores can be estimated more cheaply by using how agents work, not just whether they pass.
PTA-IRT uses historical execution trajectories as privileged calibration data, including explored context, attempted edits, and solving paths. The authors say this improves subset selection and ability estimation when evaluation budgets are low. Across four SWE benchmarks, it outperformed prior IRT baselines on score and ranking recovery. Source: ArXiv · AI/CL/LG's note.
PTA-IRT uses historical execution trajectories as privileged calibration data, including explored context, attempted edits, and solving paths. The authors say this improves subset selection and ability estimation when evaluation budgets are low. Across four SWE benchmarks, it outperformed prior IRT baselines on score and ranking recovery. Source: ArXiv · AI/CL/LG's note.
score 5