SearchHorizonBench: A Benchmark for Search Value in AI Agents
Historical estimators explained less than 2% of the variation in how much another search helped.
SearchHorizonBench contains 45,632 executions across 3,168 questions, using paired continuations and repeated runs to separate search gains from generation noise. On 1,632 held-out questions, historical calibration reduced Brier prediction error by 16.2%, mainly by improving estimates of answer success. Predicting the additional benefit of searching remained harder: search gains had a standard deviation of 21.4 percentage points. At the reference token price, free selection observations offered a 2.49-point utility opportunity over the best constant action, driven primarily by information about search gains.
HF Daily Papers' note
SearchHorizonBench contains 45,632 executions across 3,168 questions, using paired continuations and repeated runs to separate search gains from generation noise. On 1,632 held-out questions, historical calibration reduced Brier prediction error by 16.2%, mainly by improving estimates of answer success. Predicting the additional benefit of searching remained harder: search gains had a standard deviation of 21.4 percentage points. At the reference token price, free selection observations offered a 2.49-point utility opportunity over the best constant action, driven primarily by information about search gains.
HF Daily Papers' note
score 4