Explaining the Roles of Experts in Mixture-of-Experts LLMs for Personalization
Boosting specific experts shifted an unpersonalized model toward personalized behavior.
The researchers adapted three open-weight mixture-of-experts models on a large-scale personalization dataset. They compared activations before and after adaptation to identify interpretable features and the experts associated with them. The personalized models performed competitively with dense models while activating only a fraction of their parameters. Causal interventions identified distinct routing paths for different personalization features; the paper is under review.
HF Daily Papers' note
The researchers adapted three open-weight mixture-of-experts models on a large-scale personalization dataset. They compared activations before and after adaptation to identify interpretable features and the experts associated with them. The personalized models performed competitively with dense models while activating only a fraction of their parameters. Causal interventions identified distinct routing paths for different personalization features; the paper is under review.
HF Daily Papers' note
score 4