Component-Aware Differential Privacy for Federated Multilingual Speech-LLMs
The paper says common per-layer DP clipping can fail in speech-LLMs because encoder and decoder gradients live at very different scales.
The authors call the failure “cross-component budget collapse,” and report it can push WER far from flat global clipping or break training. They test six per-layer methods across three speech-LLM architectures, then propose an `alpha-split` allocation that separates encoder and LLM parameters into independent pools. They say the method keeps the original DP guarantee while recovering WER utility versus flat DP, with tighter encoder noise protection and a small LLM noise overhead. ArXiv · AI/CL/LG's note
The authors call the failure “cross-component budget collapse,” and report it can push WER far from flat global clipping or break training. They test six per-layer methods across three speech-LLM architectures, then propose an `alpha-split` allocation that separates encoder and LLM parameters into independent pools. They say the method keeps the original DP guarantee while recovering WER utility versus flat DP, with tighter encoder noise protection and a small LLM noise overhead. ArXiv · AI/CL/LG's note
score 4