Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data
Sparse MoE models lost their edge faster than dense models when training data was repeated.
The paper reports experiments from 80M to 1B active parameters, including an 8.5B-total MoE, across single- and multi-domain mixes. Dense 80M models tolerated more than 8x repetition with little degradation, while MoEs began suffering around 4x and fell behind dense models after 32x. Dropout and stronger masking-based regularization helped, with some MoEs still beating dense models past 64x repetition, but none matched all-unique data. The authors link the overfitting to early-stabilizing routing and expert specialization. ArXiv · AI/CL/LG's note
The paper reports experiments from 80M to 1B active parameters, including an 8.5B-total MoE, across single- and multi-domain mixes. Dense 80M models tolerated more than 8x repetition with little degradation, while MoEs began suffering around 4x and fell behind dense models after 32x. Dropout and stronger masking-based regularization helped, with some MoEs still beating dense models past 64x repetition, but none matched all-unique data. The authors link the overfitting to early-stabilizing routing and expert specialization. ArXiv · AI/CL/LG's note
score 5