Megadose The AI curve, compressed.

TPHC: Improving Hyper-Connections with Tensor-Parallel Grouped Updates

· HF Daily Papers ·
TPHC lets parallel residual streams receive different updates without adding parameters or FLOPs to attention or feed-forward sublayers.

The method routes a separate mixture of residual streams to each tensor-parallel shard, then keeps shard outputs separate for write-back. In a 1.1B model trained on 100B tokens, it reduced validation loss by 0.7–1.0% and improved average zero-shot accuracy across six commonsense benchmarks by 0.3–0.7 percentage points versus mHC at matched stream counts. On an 8B model with four streams, training took 9% longer than mHC. Expanding TPHC from four to sixteen streams increased training time by 19%.

HF Daily Papers' note

score 5

Categories: Research