SepGen: Multi-Stem Audio-Video Separation and Generation in a Single Model
SepGen makes each described sound source accessible as its own waveform while generating or separating audio-video scenes.
The model extends a pretrained audio-video generator to output video, a mixed soundtrack, and separate stems for captioned sources in one sampling run. In generation mode, captions define what each stem contains, including speaker-by-speaker dialogue order. In separation mode, captions tell the model what to extract from an existing clean mix, including real recordings of speech, music, and effects. The paper says SepGen beats language-conditioned separators most clearly on speech, even when spoken lines are removed from the captions. HF Daily Papers' note
The model extends a pretrained audio-video generator to output video, a mixed soundtrack, and separate stems for captioned sources in one sampling run. In generation mode, captions define what each stem contains, including speaker-by-speaker dialogue order. In separation mode, captions tell the model what to extract from an existing clean mix, including real recordings of speech, music, and effects. The paper says SepGen beats language-conditioned separators most clearly on speech, even when spoken lines are removed from the captions. HF Daily Papers' note
score 5