RARE: Decoupling Representation Steering from Expert Routing in Mixture-of-Experts Language Models
RARE tries to keep MoE expert routing stable while hidden-state steering changes model behavior.
The paper says direct representation engineering in mixture-of-experts models can disturb routing, weakening the steering effect. RARE projects behavioral perturbations into the router matrix’s null space, then corrects downstream routing drift. The authors test five perturbation estimators on six open-weight MoE models across harmfulness, truthfulness, and factual editing. Reported gains include TruthfulQA MC1 rising from 41.0% to 58.6% and CounterFact efficacy from 16.8% to 96.3%. ArXiv · AI/CL/LG's note
The paper says direct representation engineering in mixture-of-experts models can disturb routing, weakening the steering effect. RARE projects behavioral perturbations into the router matrix’s null space, then corrects downstream routing drift. The authors test five perturbation estimators on six open-weight MoE models across harmfulness, truthfulness, and factual editing. Reported gains include TruthfulQA MC1 rising from 41.0% to 58.6% and CounterFact efficacy from 16.8% to 96.3%. ArXiv · AI/CL/LG's note
score 5