EarlyEval: Cheaper Agent Evaluation via Early Outcome Prediction
EarlyEval stops agent runs once intermediate behavior is confident enough to predict the final outcome.
The paper frames this as a way to cut the cost inside each benchmark task, rather than shrinking the benchmark itself. Its framework trains LightGBM success and failure classifiers on behavioral, textual, and reference-solution features. Across SWE-bench Verified, TerminalBench, and Toolathlon, it reports eliminating 13%-26% of agent steps while keeping prediction accuracy at 89%-97%. The authors say average per-agent resolve rates moved by only one to two percentage points. ArXiv · AI/CL/LG's note
The paper frames this as a way to cut the cost inside each benchmark task, rather than shrinking the benchmark itself. Its framework trains LightGBM success and failure classifiers on behavioral, textual, and reference-solution features. Across SWE-bench Verified, TerminalBench, and Toolathlon, it reports eliminating 13%-26% of agent steps while keeping prediction accuracy at 89%-97%. The authors say average per-agent resolve rates moved by only one to two percentage points. ArXiv · AI/CL/LG's note
score 6