How Model Growth, Recursion, and Boundary Operators Influence Scaling Exponents
The paper says model-growth architectures can change pre-training scaling exponents, not just move models along the same curve.
Chen, Vegesna, Dahal, and Wilson use looped transformers as the anchor, treating extra loops during training as a way to grow usable depth. Their largest claim is that model growth, with or without shared weights, produces the biggest exponent shifts; they report a 7.4B growth architecture matching GPT-3 13B on CORE with about 20x less compute. A simpler boundary operator in a vanilla transformer also improves compute efficiency, though less strongly. In data-constrained multi-epoch training, standard looping appears useful as regularization, with more loops becoming compute-optimal at larger scale. ArXiv · AI/CL/LG's note
Chen, Vegesna, Dahal, and Wilson use looped transformers as the anchor, treating extra loops during training as a way to grow usable depth. Their largest claim is that model growth, with or without shared weights, produces the biggest exponent shifts; they report a 7.4B growth architecture matching GPT-3 13B on CORE with about 20x less compute. A simpler boundary operator in a vanilla transformer also improves compute efficiency, though less strongly. In data-constrained multi-epoch training, standard looping appears useful as regularization, with more loops becoming compute-optimal at larger scale. ArXiv · AI/CL/LG's note
score 6