ShellElastic: Scalable Elastic Pretraining with Simultaneous Maximal Updates and Hyperparameter Transfer
One pretraining run can yield multiple budget-sized transformer subnetworks at roughly the per-step cost of the widest one.
ShellElastic arranges hidden dimensions as a dense core plus shells, with block lower-triangular matrices so each narrower subnetwork appears as a prefix inside the full model’s forward pass. The paper says this keeps the training graph fixed while training all nested subnetworks jointly. It also introduces µE, scaling rules meant to preserve maximal updates and transfer hyperparameters from small proxy runs to larger elastic supernets. In reported sweeps, the resulting subnetworks beat MatFormer and fully elastic baselines at matched sizes, and come close to separately trained models that require several times more steps. HF Daily Papers' note
ShellElastic arranges hidden dimensions as a dense core plus shells, with block lower-triangular matrices so each narrower subnetwork appears as a prefix inside the full model’s forward pass. The paper says this keeps the training graph fixed while training all nested subnetworks jointly. It also introduces µE, scaling rules meant to preserve maximal updates and transfer hyperparameters from small proxy runs to larger elastic supernets. In reported sweeps, the resulting subnetworks beat MatFormer and fully elastic baselines at matched sizes, and come close to separately trained models that require several times more steps. HF Daily Papers' note
score 5