On the Design of Hyper-Connections: Simplification, Stability, and Scalability
SimpleHC matches mHC’s reported performance with just 2.4% extra training time over a vanilla residual baseline at the largest scale tested.
The researchers identify repeated access to the expanded residual state as a major source of training overhead, even without residual mixing. SimpleHC removes that mixing and shares memory access through batched reads and deferred writes. Across 15B to 30B mixture-of-experts models, it maintains performance comparable to mHC. The authors also report smoother gradient dynamics during standard training and stability under stress.
HF Daily Papers' note
The researchers identify repeated access to the expanded residual state as a major source of training overhead, even without residual mixing. SimpleHC removes that mixing and shares memory access through batched reads and deferred writes. Across 15B to 30B mixture-of-experts models, it maintains performance comparable to mHC. The authors also report smoother gradient dynamics during standard training and stability under stress.
HF Daily Papers' note
score 5