VideoRAE: Taming Video Foundation Models for Generative Modeling via Representation Autoencoders
VideoRAE turns frozen video foundation-model features into compact latents usable by both diffusion and autoregressive video generators.
The paper argues that standard 3D-VAEs can miss semantic structure because they train mainly for pixel reconstruction. VideoRAE compresses multi-scale features with a lightweight 1D self-attention projector, then supports continuous latents for Diffusion Transformers and discrete tokens for autoregressive models. Its decoder uses local-global representation alignment instead of KL regularization. In the reported tests, VideoRAE improves reconstruction, reaches UCF-101 class-conditional gFVD scores of 40 for autoregressive generation and 93 for diffusion generation, and converges about five times faster than autoencoder baselines. HF Daily Papers' note
The paper argues that standard 3D-VAEs can miss semantic structure because they train mainly for pixel reconstruction. VideoRAE compresses multi-scale features with a lightweight 1D self-attention projector, then supports continuous latents for Diffusion Transformers and discrete tokens for autoregressive models. Its decoder uses local-global representation alignment instead of KL regularization. In the reported tests, VideoRAE improves reconstruction, reaches UCF-101 class-conditional gFVD scores of 40 for autoregressive generation and 93 for diffusion generation, and converges about five times faster than autoencoder baselines. HF Daily Papers' note
score 5