LeVJEPA: Efficient & Scalable Video Pretraining without the Heuristics
LeVJEPA claims video pretraining can match or beat V-JEPA 2 while using 5.6 to 20.8 times less compute.
The paper introduces a single-encoder video model trained with LeJEPA’s collapse-free objective, avoiding EMA target encoders, stop-gradients, limited predictors, and pixel reconstruction. Its pretraining cost is tied to how many tokens the encoder sees, with uniform random token dropping both reducing compute and improving downstream accuracy. The authors also report that block-causal attention can be used without measurable accuracy loss, making temporal order part of the encoder. ArXiv · AI/CL/LG's note
The paper introduces a single-encoder video model trained with LeJEPA’s collapse-free objective, avoiding EMA target encoders, stop-gradients, limited predictors, and pixel reconstruction. Its pretraining cost is tied to how many tokens the encoder sees, with uniform random token dropping both reducing compute and improving downstream accuracy. The authors also report that block-causal attention can be used without measurable accuracy loss, making temporal order part of the encoder. ArXiv · AI/CL/LG's note
score 6