Beyond Magnitude: Contrastive Routing for Modular Mixture-of-Experts
The paper says MoE routers specialize better when tokens are scored against a shared reference state, not just by raw activation size.
CoRM compares each token with an exponential moving average of hidden states, pushing the routing signal into a more separable subspace. The authors report routing boundaries that line up more strongly with linguistic structure than a standard Top-k MoE baseline. On nine zero-shot reasoning benchmarks, they claim gains of +0.67 to +1.69 points for Top-1 and +1.38 to +1.77 for Top-2 routing. The added cost is listed as 2.9% more parameters and 2.6% more FLOPs per token. ArXiv · AI/CL/LG's note
CoRM compares each token with an exponential moving average of hidden states, pushing the routing signal into a more separable subspace. The authors report routing boundaries that line up more strongly with linguistic structure than a standard Top-k MoE baseline. On nine zero-shot reasoning benchmarks, they claim gains of +0.67 to +1.69 points for Top-1 and +1.38 to +1.77 for Top-2 routing. The added cost is listed as 2.9% more parameters and 2.6% more FLOPs per token. ArXiv · AI/CL/LG's note
score 5