SADL: On the Demand-Supply Geometry of Representation Collapse
SADL cut representation collapse from 50% to 0% across ten random seeds in the reported control experiments.
The authors find that a batch-normalized projector can hide a shared component dominating the encoder’s representations, leaving output regularization unable to prevent collapse. SADL constrains the encoder directly when it cannot preserve the differences demanded by the projector output. It retains the original training objective and adds no networks, trainable parameters or architectural changes. After one epoch, encoder effective rank was 3.4–6.4 times the baseline, while latent prediction error fell to between 1/20 and 1/303 of baseline levels.
ArXiv · AI/CL/LG's note
The authors find that a batch-normalized projector can hide a shared component dominating the encoder’s representations, leaving output regularization unable to prevent collapse. SADL constrains the encoder directly when it cannot preserve the differences demanded by the projector output. It retains the original training objective and adds no networks, trainable parameters or architectural changes. After one epoch, encoder effective rank was 3.4–6.4 times the baseline, while latent prediction error fell to between 1/20 and 1/303 of baseline levels.
ArXiv · AI/CL/LG's note
score 5