Hierarchical Residuals: A Minimalist Decoupling of Residual Width from Compute Width
HiRes reports a 1.30-fold gain in FLOPs efficiency in fitted scaling laws, while leaving individual Transformer layers unchanged.
The design groups layers into blocks, connecting them through a wider outer residual state while keeping computation inside each layer at a fixed width. Learned read/write projections bridge the two spaces. In OLMoE-1B-7B pretraining over 500 billion tokens, HiRes lowered training loss by 0.0208 and raised downstream accuracy by 1.14 points. The authors report competitive performance against VWN, mHC and AttnRes with lower activation memory and I/O costs, using standard projections and normalizations without specialized kernels.
HF Daily Papers' note
The design groups layers into blocks, connecting them through a wider outer residual state while keeping computation inside each layer at a fixed width. Learned read/write projections bridge the two spaces. In OLMoE-1B-7B pretraining over 500 billion tokens, HiRes lowered training loss by 0.0208 and raised downstream accuracy by 1.14 points. The authors report competitive performance against VWN, mHC and AttnRes with lower activation memory and I/O costs, using standard projections and normalizations without specialized kernels.
HF Daily Papers' note
score 5