Future Representation Distillation for Imagination-free World Action Models
DreamFree reports 11Hz robot control with a 14-billion-parameter video backbone, without generating future video.
The system trains a visual encoder to predict a frozen video diffusion model’s representations from the current observation, then uses those features to predict actions. This bypasses costly iterative video denoising and avoids relying on potentially inaccurate generated frames. The authors report higher success rates than state-of-the-art vision-language-action models and World Action Models across real-world and simulated tasks. They also demonstrate learning new tasks from videos without action annotations, including demonstrations from a different robot embodiment.
HF Daily Papers' note
The system trains a visual encoder to predict a frozen video diffusion model’s representations from the current observation, then uses those features to predict actions. This bypasses costly iterative video denoising and avoids relying on potentially inaccurate generated frames. The authors report higher success rates than state-of-the-art vision-language-action models and World Action Models across real-world and simulated tasks. They also demonstrate learning new tasks from videos without action annotations, including demonstrations from a different robot embodiment.
HF Daily Papers' note
score 5