LightVAE: Towards Compact and Efficient Video Autoencoders
LightVAE targets the video VAE bottleneck by compressing temporal work and network depth without discarding temporal modeling.
The paper reparameterizes causal Conv3D layers into a dense spatial response plus a difference-driven temporal correction, then applies low-rank compression only to that temporal correction. It also prunes blocks while preserving width, using a frozen teacher suffix to guide feature alignment after compression. Across three video VAE backbones and two GPU platforms, the authors report faster decoding, including up to 2.97x original throughput on MiniMax-H3 while retaining 99.51% of the original VBench-2.0 score. Code is listed as available. HF Daily Papers' note
The paper reparameterizes causal Conv3D layers into a dense spatial response plus a difference-driven temporal correction, then applies low-rank compression only to that temporal correction. It also prunes blocks while preserving width, using a frozen teacher suffix to guide feature alignment after compression. Across three video VAE backbones and two GPU platforms, the authors report faster decoding, including up to 2.97x original throughput on MiniMax-H3 while retaining 99.51% of the original VBench-2.0 score. Code is listed as available. HF Daily Papers' note
score 4