Dreaming the Sound of Contact: Leveraging Video and Audio Generation for Zero-Shot Force-Aware Manipulation and Data Generation
The paper turns generated contact sounds into force targets for robot manipulation.
The authors propose using video generation for motion and generated audio loudness for a bounded, time-varying desired-force profile. A Franka Panda robot then follows those trajectories with a closed-loop force regulator during contact. In their tests, the force-aware setup succeeds on multiple contact-heavy tasks where a kinematic-only baseline fails. They also use the same pipeline to generate data for training closed-loop policies. ArXiv · AI/CL/LG's note
The authors propose using video generation for motion and generated audio loudness for a bounded, time-varying desired-force profile. A Franka Panda robot then follows those trajectories with a closed-loop force regulator during contact. In their tests, the force-aware setup succeeds on multiple contact-heavy tasks where a kinematic-only baseline fails. They also use the same pipeline to generate data for training closed-loop policies. ArXiv · AI/CL/LG's note
score 5