LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering
The benchmark separates a model’s control decisions from the coding agent doing the work.
LoopArena tests a “Controller” model that reads structured progress from a fixed “Worker” coding agent and decides what to do next, what to verify, or when to stop. The paper frames this as a way to measure loop guidance itself, rather than the Worker’s raw coding ability. Its three evaluation modes range from execution-validated next-step choices to full task runs. On full tasks, the best reported Strict Success Rate is 24.69%, with average estimated inference-cost reductions of 64.4% across Controllers. HF Daily Papers' note
LoopArena tests a “Controller” model that reads structured progress from a fixed “Worker” coding agent and decides what to do next, what to verify, or when to stop. The paper frames this as a way to measure loop guidance itself, rather than the Worker’s raw coding ability. Its three evaluation modes range from execution-validated next-step choices to full task runs. On full tasks, the best reported Strict Success Rate is 24.69%, with average estimated inference-cost reductions of 64.4% across Controllers. HF Daily Papers' note
score 5