Contrastive latent representation learning with denoising score matching
ScoreCLR makes the repulsion cost linear in batch size, with minimal loss of linear-probe accuracy in the reported tests.
Contrastive learning typically pushes different samples apart using explicit negative examples, an approach that can require large batches and produce high-variance estimates. ScoreCLR instead learns that repulsion through a lightweight network trained alongside the encoder using denoising score matching. On CIFAR-10 and ImageNet-100, the authors report accuracy close to SimCLR under a shared training recipe. Its projector outputs also have higher effective rank and sit closer to the Gaussian prior used for regularization.
ArXiv · AI/CL/LG's note
Contrastive learning typically pushes different samples apart using explicit negative examples, an approach that can require large batches and produce high-variance estimates. ScoreCLR instead learns that repulsion through a lightweight network trained alongside the encoder using denoising score matching. On CIFAR-10 and ImageNet-100, the authors report accuracy close to SimCLR under a shared training recipe. Its projector outputs also have higher effective rank and sit closer to the Gaussian prior used for regularization.
ArXiv · AI/CL/LG's note
score 5