Sharpness-Aware Minimization and Muon: Robustness under the Spectral Norm
A spectral-norm SAM perturbation paired with Muon delivered the strongest ImageNet results in the paper’s tests.
The authors study SAM’s sensitivity to the geometry used to define “small” parameter changes. They introduce a layerwise spectral inner perturbation for matrix-valued hidden-layer weights, then test it with AdamW/SGDW or Muon as the outer optimizer. On ImageNet-1K with ViT-Small/16 and ResNet-50, the spectral inner step plus Muon outer step had the best validation accuracy among the evaluated methods. Source: ArXiv · AI/CL/LG's note.
The authors study SAM’s sensitivity to the geometry used to define “small” parameter changes. They introduce a layerwise spectral inner perturbation for matrix-valued hidden-layer weights, then test it with AdamW/SGDW or Muon as the outer optimizer. On ImageNet-1K with ViT-Small/16 and ResNet-50, the spectral inner step plus Muon outer step had the best validation accuracy among the evaluated methods. Source: ArXiv · AI/CL/LG's note.
score 4