Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing
The paper argues that MoE experts can be geometrically similar and still add useful computation.
Across six MoE architectures, the authors find substantial overlap among expert subspaces, but actual routing choices still explain token representations better than matched alternatives. In OLMoE, Mixtral, and DeepSeek tests, selected experts beat the strongest unselected rivals, while the real prefix consistently narrows that edge. Frozen-route comparisons show later experts improve next-token prediction in 24 of 39 cases, with the rest inconclusive. The authors call the pattern “coherent overlap,” meaning routing draws from a shared neighborhood rather than cleanly separated expert directions. Source: HF Daily Papers' note
Across six MoE architectures, the authors find substantial overlap among expert subspaces, but actual routing choices still explain token representations better than matched alternatives. In OLMoE, Mixtral, and DeepSeek tests, selected experts beat the strongest unselected rivals, while the real prefix consistently narrows that edge. Frozen-route comparisons show later experts improve next-token prediction in 24 of 39 cases, with the rest inconclusive. The authors call the pattern “coherent overlap,” meaning routing draws from a shared neighborhood rather than cleanly separated expert directions. Source: HF Daily Papers' note
score 4