Improving Environment-Agnostic Coding Agents: The Role of Scaffolding
An instruction to report difficulties and hand back control improved scores on two benchmarks, but gains came with higher token costs.
The agent’s code, prompts and general-purpose tools stayed unchanged across environments, apart from the instruction being tested. Mean scores rose on ARC-AGI-3 and AGI Maze, while Terminal-Bench results showed small, mixed changes. A scripted reset controller recovered part of the gap with an ARC-specific reference agent, suggesting that supplied procedures still helped the tested models. The experiments could not separate better use of information and procedures from additional or differently allocated inference.
ArXiv · AI/CL/LG's note
The agent’s code, prompts and general-purpose tools stayed unchanged across environments, apart from the instruction being tested. Mean scores rose on ARC-AGI-3 and AGI Maze, while Terminal-Bench results showed small, mixed changes. A scripted reset controller recovered part of the gap with an ARC-specific reference agent, suggesting that supplied procedures still helped the tested models. The experiments could not separate better use of information and procedures from additional or differently allocated inference.
ArXiv · AI/CL/LG's note
score 4