Learning the Task or Learning the Verifier? Rethinking Criteria Exposure in Embodied Self-Improvement
Giving an embodied coding agent the benchmark’s success criteria raised its scores, but slight changes to those criteria erased the advantage.
The controlled experiments changed the checks while keeping them consistent with the original task instructions. Skills developed with criteria exposure also transferred poorly across tasks. The authors argue that existing checks can demand more than the instructions require, distorting assessments of agent-written robot programs. They propose instruction-faithful criteria and report that these better reflect program performance.
HF Daily Papers' note
The controlled experiments changed the checks while keeping them consistent with the original task instructions. Skills developed with criteria exposure also transferred poorly across tasks. The authors argue that existing checks can demand more than the instructions require, distorting assessments of agent-written robot programs. They propose instruction-faithful criteria and report that these better reflect program performance.
HF Daily Papers' note
score 5