In-Context Robot Learning with VLM Agents
GPT-Policy tests whether commercial VLMs can turn deployment-time examples and feedback into robot actions without retraining.
The paper introduces a framework that compiles task-relevant visual context, has a VLM propose robot-tool actions, and uses a constrained controller to verify and execute them. In real-robot trials, human video demonstrations improved task completion even without robot action labels. Action-aligned references helped further on contact-sensitive tasks. The authors frame the result as an empirical step toward in-context robot adaptation, while emphasizing reliability limits still remain. HF Daily Papers' note
The paper introduces a framework that compiles task-relevant visual context, has a VLM propose robot-tool actions, and uses a constrained controller to verify and execute them. In real-robot trials, human video demonstrations improved task completion even without robot action labels. Action-aligned references helped further on contact-sensitive tasks. The authors frame the result as an empirical step toward in-context robot adaptation, while emphasizing reliability limits still remain. HF Daily Papers' note
score 5