SciResearch-Eval: Evaluating Scientific Judgment in Autonomous Experimental Agents
Most models’ claimed 95% uncertainty ranges contained the truth just 20–62% of the time when their simulator lacked relevant physics.
SciResearch-Eval asks agents to design heating experiments, estimate material properties and report uncertainty while using a potentially flawed simulator. Across 93 held-out tasks and 2,511 episodes, GPT-6 Astra achieved 94% coverage and Claude Fable 5.2 reached 99%, partly by choosing experiments where the simulator remained accurate. Asking for wider intervals barely changed coverage. Maximum reasoning effort improved Claude Opus 5.5’s interval score more than tenfold, bringing it level with Astra at default effort, but required 13–31 times more reasoning.
ArXiv · AI/CL/LG's note
SciResearch-Eval asks agents to design heating experiments, estimate material properties and report uncertainty while using a potentially flawed simulator. Across 93 held-out tasks and 2,511 episodes, GPT-6 Astra achieved 94% coverage and Claude Fable 5.2 reached 99%, partly by choosing experiments where the simulator remained accurate. Asking for wider intervals barely changed coverage. Maximum reasoning effort improved Claude Opus 5.5’s interval score more than tenfold, bringing it level with Astra at default effort, but required 13–31 times more reasoning.
ArXiv · AI/CL/LG's note
score 5