The Loss Does Not See the Basis, but Adam Does
Adam can pick a different solution even when the loss treats two parameter bases as equivalent.
The paper argues that gradient descent keeps a low-rank bias in factored matrix models because its updates respect the model’s gauge symmetry, while Adam’s coordinate-wise preconditioning breaks that symmetry. It sorts optimizers by whether they preserve that equivariance, placing Adam and RMSProp outside the group that can inherit gradient flow’s low-rank behavior. Experiments include matrix sensing, transformers, and hyperspectral data, where basis-sensitive updates lead to different invariants or worse held-out error at matched loss. Source: HF Daily Papers' note.
The paper argues that gradient descent keeps a low-rank bias in factored matrix models because its updates respect the model’s gauge symmetry, while Adam’s coordinate-wise preconditioning breaks that symmetry. It sorts optimizers by whether they preserve that equivariance, placing Adam and RMSProp outside the group that can inherit gradient flow’s low-rank behavior. Experiments include matrix sensing, transformers, and hyperspectral data, where basis-sensitive updates lead to different invariants or worse held-out error at matched loss. Source: HF Daily Papers' note.
score 5