Multi-Stream Attention Residuals
MSAR turns residual connections into multi-head depth attention, reporting a 1.58x iso-loss compute speedup over standard residuals.
The paper extends Attention Residuals by letting different heads retrieve past layer representations through separate contextual views. It uses multi-scale causal convolutions for stream diversity, then mixes the retrieved streams back into a single layer input. The authors report gains across 12 downstream benchmarks at 18B and 28B MoE scales, beating standard residuals, mHC, and AttnRes. HF Daily Papers' note
The paper extends Attention Residuals by letting different heads retrieve past layer representations through separate contextual views. It uses multi-scale causal convolutions for stream diversity, then mixes the retrieved streams back into a single layer input. The authors report gains across 12 downstream benchmarks at 18B and 28B MoE scales, beating standard residuals, mHC, and AttnRes. HF Daily Papers' note
score 5