RoboTryOn: Synthesizing Dexterous Robot Demonstrations from Human Videos via Virtual Try-On
Policies trained on synthetic demonstrations reached 60% average success on new tasks, up from 2%, without real robot demonstrations of those tasks.
RoboTryOn replaces the human in a video with a target robot while preserving the demonstrated motion, objects and background. Human hand skeletons guide video synthesis, and a depth-aware inverse dynamics model infers actions for policy training. The authors report results across multiple humanoid embodiments, including difficult two-handed manipulation. Training with the synthetic data also matched policies trained solely on teleoperated demonstrations at roughly one-third of the collection cost.
HF Daily Papers' note
RoboTryOn replaces the human in a video with a target robot while preserving the demonstrated motion, objects and background. Human hand skeletons guide video synthesis, and a depth-aware inverse dynamics model infers actions for policy training. The authors report results across multiple humanoid embodiments, including difficult two-handed manipulation. Training with the synthetic data also matched policies trained solely on teleoperated demonstrations at roughly one-third of the collection cost.
HF Daily Papers' note
score 5