RubatoCodec: Budgeted Adaptive Speech Tokenization with Streaming Decoding
RubatoCodec fixes the number of speech events per utterance while adapting how much audio each event covers.
Built on X-Codec-2, it detects changes in codec features and waveform spectra to allocate events without a learned boundary predictor. Each event carries discrete codes and an explicit duration, enabling decoding one event at a time. The authors report better reconstruction fidelity and intelligibility than fixed-rate coding at low event rates. In a matched text-to-speech comparison, the adaptive representation reduced word error rates and improved UTMOS quality scores while maintaining comparable speaker similarity.
HF Daily Papers' note
Built on X-Codec-2, it detects changes in codec features and waveform spectra to allocate events without a learned boundary predictor. Each event carries discrete codes and an explicit duration, enabling decoding one event at a time. The authors report better reconstruction fidelity and intelligibility than fixed-rate coding at low event rates. In a matched text-to-speech comparison, the adaptive representation reduced word error rates and improved UTMOS quality scores while maintaining comparable speaker similarity.
HF Daily Papers' note
score 5