Megadose The AI curve, compressed.

Loopback Transformer: Towards a Unified View of Residual Looped Transformers

· HF Daily Papers ·
Loopback reports lower validation loss using 40–75% of the Transformer parameters of standard baselines.

On ClimbMix, the comparisons matched training tokens and executed depth at reference scales from 191M to 3B; parameter counts exclude token embeddings and the output head. The design keeps the shared stack’s input separate from individual updates, using geometric decay to set their total weight and attention to distribute it. It also achieved higher mean accuracy across thirteen downstream tasks at 1.4B and 3B. At 191M, 374M and 778M, it led both measures against the standard Transformer and seven looped residual designs.

HF Daily Papers' note

score 5

Categories: Research