Dion3: Full-Stack Orthogonal Updates
Dion3 keeps Muon’s loss benefits while cutting optimizer step time by up to 6x.
The paper says Muon’s Newton-Schulz orthogonalization is costly, especially when sharded weights add communication overhead. Dion3 attacks that with a cheaper Gram Newton-Schulz method, symmetry-aware CuteDSL kernels, and megabatching. It also orthogonalizes only a fraction of the momentum matrix rows each step, which the authors say improves on Dion in speed and performance. The package is presented as a drop-in Muon replacement. HF Daily Papers' note
The paper says Muon’s Newton-Schulz orthogonalization is costly, especially when sharded weights add communication overhead. Dion3 attacks that with a cheaper Gram Newton-Schulz method, symmetry-aware CuteDSL kernels, and megabatching. It also orthogonalizes only a fraction of the momentum matrix rows each step, which the authors say improves on Dion in speed and performance. The package is presented as a drop-in Muon replacement. HF Daily Papers' note
score 5