Proving and Improving the Platonic Representation Hypothesis with JEPAs
A text JEPA reached the same cross-modal alignment with 42 times fewer training samples than next-token prediction on the same corpus.
Balestriero and LeCun prove that separately trained JEPAs learn representations equivalent up to a rotation when their positive pairs are observations of the same scene. Their experiments span 145,000 captions to text from 940 million web images. A 109-million-parameter text JEPA aligned with DINOv2 and CLIP image encoders better than every language model evaluated, including models up to 72 billion parameters. The results identify the choice of paired observations as a lever for alignment, rather than scale alone.
HF Daily Papers' note
Balestriero and LeCun prove that separately trained JEPAs learn representations equivalent up to a rotation when their positive pairs are observations of the same scene. Their experiments span 145,000 captions to text from 940 million web images. A 109-million-parameter text JEPA aligned with DINOv2 and CLIP image encoders better than every language model evaluated, including models up to 72 billion parameters. The results identify the choice of paired observations as a lever for alignment, rather than scale alone.
HF Daily Papers' note
score 5