ARE-bench: Benchmarking Agents' Research Ability in Real-World Researcher Sessions and Workspaces
The strongest model met every requirement on just 66.2% of tasks drawn from ongoing research projects.
ARE-bench tests 75 tasks using researchers’ actual agent sessions and unfinished workspaces, preserving earlier decisions, outdated files and failed attempts. Seven frontier models were evaluated, with Design and Execution among the hardest stages for all seven. Three models passed about 69% of tasks when checked only for delivery contracts and general norms; their pass rates fell to 28.4%–40.0% when prior instructions and material or workspace requirements also counted. The authors argue that omitting those contextual checks can overestimate research agents’ abilities.
ArXiv · AI/CL/LG's note
ARE-bench tests 75 tasks using researchers’ actual agent sessions and unfinished workspaces, preserving earlier decisions, outdated files and failed attempts. Seven frontier models were evaluated, with Design and Execution among the hardest stages for all seven. Three models passed about 69% of tasks when checked only for delivery contracts and general norms; their pass rates fell to 28.4%–40.0% when prior instructions and material or workspace requirements also counted. The authors argue that omitting those contextual checks can overestimate research agents’ abilities.
ArXiv · AI/CL/LG's note
score 5