Proving and Improving the Platonic Representation Hypothesis with JEPAs
The authors report that well-chosen training pairs reach the same cross-modal alignment with 42 times fewer samples than next-token prediction.
Balestriero and LeCun prove that separately trained JEPAs learn matching representations, up to a single rotation, when their positive pairs are two observations of the same scene. Their analysis treats next-token prediction as a JEPA with poorer views drawn from parts of one document. In their experiments, a 109M-parameter text JEPA aligned with image encoders better than every language model they evaluated, including models up to 72B parameters. Tests ranged from 145,000 captions to the texts associated with 940 million web images.
ArXiv · AI/CL/LG's note
Balestriero and LeCun prove that separately trained JEPAs learn matching representations, up to a single rotation, when their positive pairs are two observations of the same scene. Their analysis treats next-token prediction as a JEPA with poorer views drawn from parts of one document. In their experiments, a 109M-parameter text JEPA aligned with image encoders better than every language model they evaluated, including models up to 72B parameters. Tests ranged from 145,000 captions to the texts associated with 940 million web images.
ArXiv · AI/CL/LG's note
score 5