The Life Cycle of a Massive Activation: Stochastic Birth, Weight-Decay-Driven Growth, and Competitive Consolidation
Weight decay is presented as the causal control on how massive activations rise, peak, and fade during transformer training.
The paper traces “sink-carrying” residual-stream channels from early emergence through later consolidation into a smaller set of carriers. It says the relevant scale is collective: gradient attenuation tracks the sink token’s root-mean-square magnitude across channels, not one channel alone. In controlled runs, removing weight decay near the peak lets global activation scale keep rising, while keeping it causes decline even without changing the learning rate. The authors model the peak as a balance between AdamW-preconditioned growth and decay, with decay strength shifting peak timing and magnitude. HF Daily Papers' note
The paper traces “sink-carrying” residual-stream channels from early emergence through later consolidation into a smaller set of carriers. It says the relevant scale is collective: gradient attenuation tracks the sink token’s root-mean-square magnitude across channels, not one channel alone. In controlled runs, removing weight decay near the peak lets global activation scale keep rising, while keeping it causes decline even without changing the learning rate. The authors model the peak as a balance between AdamW-preconditioned growth and decay, with decay strength shifting peak timing and magnitude. HF Daily Papers' note
score 4