VidDub: Replacement-Aware Distillation for Two-Step Video Dubbing
Ignoring the source mouth during replacement-audio training improved lip synchronization, with the final editor distilled to two denoising steps.
VidDub trains replacement mouth movements against audio-visual synchronization while preserving the surrounding face and background. Across three training seeds, removing conflicting source-mouth reconstruction losses improved cross-audio Sync-C by 0.833. The authors report strong lip synchronization and superior visual quality, identity preservation, and visual speech recognition across four benchmarks. They plan to release the curated training dataset, VidDub-Bench, evaluation manifests, preprocessing outputs, and evaluation code.
HF Daily Papers' note
VidDub trains replacement mouth movements against audio-visual synchronization while preserving the surrounding face and background. Across three training seeds, removing conflicting source-mouth reconstruction losses improved cross-audio Sync-C by 0.833. The authors report strong lip synchronization and superior visual quality, identity preservation, and visual speech recognition across four benchmarks. They plan to release the curated training dataset, VidDub-Bench, evaluation manifests, preprocessing outputs, and evaluation code.
HF Daily Papers' note
score 4