Randomized Momentum Transport with Cheap Predictors
The paper proposes skipping most momentum transport corrections while keeping the estimator unbiased.
Chayti and Jaggi introduce a randomized transport step that computes the expensive correction only with probability `q`, using a cheaper predictor otherwise. The method preserves the conditional mean, but adds variance tied to the predictor’s loss. Their analysis keeps an `O(T^{-1/3})` stationarity bound for fixed `q > 0` under stated assumptions and covers LMO directions including normalized, sign, and exact spectral Muon updates. In experiments at roughly equal oracle cost, using one quarter of the corrections reduced MNIST1D final training loss by 18% to 43% and improved CIFAR-10 Hessian-vector transport test accuracy by 1.5 points. ArXiv · AI/CL/LG's note
Chayti and Jaggi introduce a randomized transport step that computes the expensive correction only with probability `q`, using a cheaper predictor otherwise. The method preserves the conditional mean, but adds variance tied to the predictor’s loss. Their analysis keeps an `O(T^{-1/3})` stationarity bound for fixed `q > 0` under stated assumptions and covers LMO directions including normalized, sign, and exact spectral Muon updates. In experiments at roughly equal oracle cost, using one quarter of the corrections reduced MNIST1D final training loss by 18% to 43% and improved CIFAR-10 Hessian-vector transport test accuracy by 1.5 points. ArXiv · AI/CL/LG's note
score 4