DeepLoop: Depth Scaling for Looped Transformers
Looped Transformers need residual scaling that accounts for repeated visits to the same parameters.
The paper formalizes how reused blocks change gradient behavior through a visit-alignment coefficient, `κR`. In the conservative aligned case, it says the scaling exponent should rise from `1/4` to `1/2` as loop count grows at fixed physical depth. DeepLoop keeps the Post-LN DeepNorm setup and sets `α=(2N)^{1/2}` and `β=(8N)^{-1/2}` for unrolled depth `N`. In GPT-style looped models at GPT-2 small and medium scale, it reports no penalty without revisits and better validation loss and downstream accuracy once recurrent depth is used. HF Daily Papers' note
The paper formalizes how reused blocks change gradient behavior through a visit-alignment coefficient, `κR`. In the conservative aligned case, it says the scaling exponent should rise from `1/4` to `1/2` as loop count grows at fixed physical depth. DeepLoop keeps the Post-LN DeepNorm setup and sets `α=(2N)^{1/2}` and `β=(8N)^{-1/2}` for unrolled depth `N`. In GPT-style looped models at GPT-2 small and medium scale, it reports no penalty without revisits and better validation loss and downstream accuracy once recurrent depth is used. HF Daily Papers' note
score 4