xHC: Expanded Hyper-Connections
xHC pushes Hyper-Connections past the usual N=4 limit by updating only 4 of 16 residual streams.
The paper says earlier HC methods hit diminishing gains and rising training cost beyond N=4. xHC adds temporal feature augmentation and a sparse residual-stream design while keeping dense access to the full residual state. In 18B and 28B MoE tests, it reports consistent downstream gains, including a 4.0-point average improvement over mHC on an 18B MoE model. The authors also introduce xHC-Flash to cut memory traffic while retaining xHC’s gains. HF Daily Papers' note
The paper says earlier HC methods hit diminishing gains and rising training cost beyond N=4. xHC adds temporal feature augmentation and a sparse residual-stream design while keeping dense access to the full residual state. In 18B and 28B MoE tests, it reports consistent downstream gains, including a 4.0-point average improvement over mHC on an 18B MoE model. The authors also introduce xHC-Flash to cut memory traffic while retaining xHC’s gains. HF Daily Papers' note
score 5