Same Simulation, Different Evidence: Where Retrieved Tool Output Costs LLM Agents Accuracy
GPT agents often computed the right value, then chose the wrong answer using evidence from their own files.
Across 158 verified geochemistry questions, retrieving simulator reports through a section index cut GPT-5.1 accuracy by 11.3 percentage points and GPT-5.2 by 7.9 points; neither Claude model showed a detectable loss. Replaying 11,007 simulator calls showed the GPT models still computed the correct values about as often, but often skipped the reports and misrounded higher-precision values from their own output files. GPT-5.4’s improved simulations masked an evidence-use loss of about ten points, leaving no detectable net change. Explicitly naming the authoritative output recovered 6.3–7.9 points across the three GPT models; instructions to round correctly did not.
HF Daily Papers' note
Across 158 verified geochemistry questions, retrieving simulator reports through a section index cut GPT-5.1 accuracy by 11.3 percentage points and GPT-5.2 by 7.9 points; neither Claude model showed a detectable loss. Replaying 11,007 simulator calls showed the GPT models still computed the correct values about as often, but often skipped the reports and misrounded higher-precision values from their own output files. GPT-5.4’s improved simulations masked an evidence-use loss of about ten points, leaving no detectable net change. Explicitly naming the authoritative output recovered 6.3–7.9 points across the three GPT models; instructions to round correctly did not.
HF Daily Papers' note
score 5