DexForesight: Grounding Dexterous Policies with Implicit Spatial Foresight
DexForesight raised simulated task success from 49.4% to 60.5% without requiring depth input at deployment.
During training, its world model learns from scene depth, hand geometry and demonstrations to predict hand-point motion and visual changes, including movement of fingers hidden from cameras. Those learned dynamics are transferred into the policy, which runs using RGB images, language and robot state without explicitly predicting the future. The simulation result covers 11 DexJoCo tasks. Across three real-robot tasks, the method achieved 58.3% success versus 50.0% for the strongest baseline.
HF Daily Papers' note
During training, its world model learns from scene depth, hand geometry and demonstrations to predict hand-point motion and visual changes, including movement of fingers hidden from cameras. Those learned dynamics are transferred into the policy, which runs using RGB images, language and robot state without explicitly predicting the future. The simulation result covers 11 DexJoCo tasks. Across three real-robot tasks, the method achieved 58.3% success versus 50.0% for the strongest baseline.
HF Daily Papers' note
score 4