TPHC: Improving Hyper-Connections with Tensor-Parallel Grouped Updates
TPHC lets parallel residual streams receive different updates without adding parameters or FLOPs to attention or feed-forward sublayers.
The method routes a separate mixture of residual streams to each tensor-parallel shard, then keeps shard outputs separate for write-back. In a 1.1B model trained on 100B tokens, it reduced validation loss by 0.7–1.0% and improved average zero-shot accuracy across six commonsense benchmarks by 0.3–0.7 percentage points versus mHC at matched stream counts. On an 8B model with four streams, training took 9% longer than mHC. Expanding TPHC from four to sixteen streams increased training time by 19%.
HF Daily Papers' note
The method routes a separate mixture of residual streams to each tensor-parallel shard, then keeps shard outputs separate for write-back. In a 1.1B model trained on 100B tokens, it reduced validation loss by 0.7–1.0% and improved average zero-shot accuracy across six commonsense benchmarks by 0.3–0.7 percentage points versus mHC at matched stream counts. On an 8B model with four streams, training took 9% longer than mHC. Expanding TPHC from four to sixteen streams increased training time by 19%.
HF Daily Papers' note
score 5