oHC: Orthogonal Hyper-Connections on SO(4) via Quaternions
The paper argues that keeping Transformer hyper-connection mixing strictly orthogonal preserves stream diversity without destabilizing training.
The authors say unconstrained Hyper-Connections can amplify residual streams across layers, while doubly stochastic constraints prevent amplification but still let stream differences shrink. Their proposed oHC instead restricts the mixing matrix to `SO(n)`, so it neither amplifies nor attenuates any direction. For the four-stream case, they parameterize the rotation with two unit quaternions, avoiding extra parameters and iterative projection. They report better downstream results than single-stream residuals, mHC, and identity-fixed iHC. ArXiv · AI/CL/LG's note
The authors say unconstrained Hyper-Connections can amplify residual streams across layers, while doubly stochastic constraints prevent amplification but still let stream differences shrink. Their proposed oHC instead restricts the mixing matrix to `SO(n)`, so it neither amplifies nor attenuates any direction. For the four-stream case, they parameterize the rotation with two unit quaternions, avoiding extra parameters and iterative projection. They report better downstream results than single-stream residuals, mHC, and identity-fixed iHC. ArXiv · AI/CL/LG's note
score 5