Multi-Head Attention Residuals
The paper’s claim is that splitting residual-depth routing by feature subspace improves Transformer training at scale.
MHAR gives each subspace head its own softmax over layer history, adding no parameters and little compute. In scratch training, it beat a standard Transformer at 100M, 350M, and 1B parameters, with larger gains at the bigger sizes. The authors report a U-shaped result for head count, with H=4 or H=8 working best and H=16 giving some gains back. They also describe an 8B mid-training conversion that improved GSM8K by 3.2 points and GPQA by 3.1. HF Daily Papers' note
MHAR gives each subspace head its own softmax over layer history, adding no parameters and little compute. In scratch training, it beat a standard Transformer at 100M, 350M, and 1B parameters, with larger gains at the bigger sizes. The authors report a U-shaped result for head count, with H=4 or H=8 working best and H=16 giving some gains back. They also describe an 8B mid-training conversion that improved GSM8K by 3.2 points and GPQA by 3.1. HF Daily Papers' note
score 5