Routing Drift Alone Does Not Diagnose Failure in Merged MoE LLMs
Expert rerouting after MoE model merging mostly came from shifted inputs, and did not reliably predict recoverable task loss.
The paper tests merged DeepSeekMoE, OLMoE, and Qwen3-MoE models with counterfactual routing interventions. It finds that restoring source-model routes often fails to produce reliable next-token likelihood gains. The authors define routing failure as loss that can be recovered by a routing intervention while non-routing weights stay fixed. Their Selective Router Repair case study also fails to make source-informed local fixes dependable. HF Daily Papers' note
The paper tests merged DeepSeekMoE, OLMoE, and Qwen3-MoE models with counterfactual routing interventions. It finds that restoring source-model routes often fails to produce reliable next-token likelihood gains. The authors define routing failure as loss that can be recovered by a routing intervention while non-routing weights stay fixed. Their Selective Router Repair case study also fails to make source-informed local fixes dependable. HF Daily Papers' note
score 4