Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depth
The paper argues that Transformer depth fails or holds together depending on how architecture preserves Jacobian rank at initialization.
Katie Everett reframes skip connections and normalization as rank-preservation devices, not just magnitude controls. The abstract says Post-Norm rank collapses while Pre-Norm plateaus because normalization placement changes the branch-to-skip ratio across depth. It also credits the two-matrix feedforward structure and width expansion with preventing representation collapse and keeping the branch Jacobian full rank. The paper reports that initialization rank of the input-output Jacobian predicts which networks train on CIFAR-10. ArXiv · AI/CL/LG's note
Katie Everett reframes skip connections and normalization as rank-preservation devices, not just magnitude controls. The abstract says Post-Norm rank collapses while Pre-Norm plateaus because normalization placement changes the branch-to-skip ratio across depth. It also credits the two-matrix feedforward structure and width expansion with preventing representation collapse and keeping the branch Jacobian full rank. The paper reports that initialization rank of the input-output Jacobian predicts which networks train on CIFAR-10. ArXiv · AI/CL/LG's note
score 4