Six Layers Less: Encoder Pruning for Whisper with Label-Free Recovery
The paper cuts six Whisper encoder layers without needing custom inference code.
The authors rank encoder layers by how little Word Error Rate changes when each one is removed. They prune the six least costly layers, or 18.5% of the encoder stack. Label-free distillation on unlabeled monolingual speech reduces the pruning hit: mean WER across four languages is 20.1%, versus 21.9% after zero-shot pruning and 18.2% for the baseline. They say they are releasing both code and the pruned model. HF Daily Papers' note
The authors rank encoder layers by how little Word Error Rate changes when each one is removed. They prune the six least costly layers, or 18.5% of the encoder stack. Label-free distillation on unlabeled monolingual speech reduces the pruning hit: mean WER across four languages is 20.1%, versus 21.9% after zero-shot pruning and 18.2% for the baseline. They say they are releasing both code and the pruned model. HF Daily Papers' note
score 4