Actor, Judge, or Teacher? Understanding Decision Models in Long-Horizon Agents
JEV lifted a small policy’s success rate from 72.8% to 82.9% on ALFWorld and from 56.8% to 71.4% on WebShop.
The researchers tested whether models that struggle to choose actions can still provide useful training feedback. All six models tested as actors were distracted by an easier subgoal before completing a prerequisite. Given the full trajectory afterward, those same models recognized progress toward the prerequisite; asking whether each action helped also distinguished useful actions better than asking whether the task would succeed. The results suggest that seeing an episode’s consequences can make a poor actor an effective teacher.
HF Daily Papers' note
The researchers tested whether models that struggle to choose actions can still provide useful training feedback. All six models tested as actors were distracted by an easier subgoal before completing a prerequisite. Given the full trajectory afterward, those same models recognized progress toward the prerequisite; asking whether each action helped also distinguished useful actions better than asking whether the task would succeed. The results suggest that seeing an episode’s consequences can make a poor actor an effective teacher.
HF Daily Papers' note
score 5