Latent-Foresight: End-to-End Learning Predictable Representations for Latent World Models
Latent-Foresight trains the representation and the future predictor together, aiming to make the latent space itself easier to forecast.
The paper argues that prior latent world-model pipelines freeze a compressed VFM feature space before training a separate dynamics predictor. Its proposed framework jointly learns a latent tokenizer and a flow-based generative dynamics model to reduce that mismatch. The authors say design choices were added to avoid latent collapse and align reconstruction with prediction objectives. In experiments, they report more temporally coherent latents and stronger results than two-stage baselines across future scene understanding tasks and horizons. HF Daily Papers' note
The paper argues that prior latent world-model pipelines freeze a compressed VFM feature space before training a separate dynamics predictor. Its proposed framework jointly learns a latent tokenizer and a flow-based generative dynamics model to reduce that mismatch. The authors say design choices were added to avoid latent collapse and align reconstruction with prediction objectives. In experiments, they report more temporally coherent latents and stronger results than two-stage baselines across future scene understanding tasks and horizons. HF Daily Papers' note
score 4