A Spectral Theory of Grokking: Weight Decay induces Feature Learning
The paper argues that weight decay can drive the post-memorization feature learning that makes grokking appear late.
The authors model a shift from lazy NTK behavior to evolving, task-aligned kernel directions after training accuracy has already saturated. Their reduced spectral system predicts grokking time from the product of learning rate and weight decay, with a critical decay beyond which generalization fails. Tests on modular addition in a homogeneous MLP and a one-block Transformer recover the predicted phase structure and timing scale. ArXiv · AI/CL/LG's note
The authors model a shift from lazy NTK behavior to evolving, task-aligned kernel directions after training accuracy has already saturated. Their reduced spectral system predicts grokking time from the product of learning rate and weight decay, with a critical decay beyond which generalization fails. Tests on modular addition in a homogeneous MLP and a one-block Transformer recover the predicted phase structure and timing scale. ArXiv · AI/CL/LG's note
score 5