Log-Depth Recurrent Language Modeling
The paper proposes an autoregressive recurrent LM that computes every prefix representation in logarithmic depth.
It adapts balanced-tree recursive operators from sequence encoding to prediction, aiming to avoid Transformers’ fixed depth and quadratic token cost. The authors report linear runtime, robust length extrapolation, and performance close to ALiBi-based Transformers in their initial experiments. They frame it as an early characterization of a possible alternative language-modeling architecture. ArXiv · AI/CL/LG's note
It adapts balanced-tree recursive operators from sequence encoding to prediction, aiming to avoid Transformers’ fixed depth and quadratic token cost. The authors report linear runtime, robust length extrapolation, and performance close to ALiBi-based Transformers in their initial experiments. They frame it as an early characterization of a possible alternative language-modeling architecture. ArXiv · AI/CL/LG's note
score 5