V-JEPA-LM: Video-Language Models That Read Dynamics Beyond Static, Text-Aligned Representations
V-JEPA encoded speed well enough for 87% accuracy, yet the attached language model answered at chance.
The authors trace the gap to caption and question-answer training that rewards retrieving named attributes while leaving motion information unused. They train V-JEPA-LM on transformed video variants whose answers depend on changes in dynamics. At a matched training budget, this supervision adds 4.5 points on average and 38 points on speed questions in natural benchmarks. The authors report that their 4B model averages 72.1 across eleven video benchmarks, against 61.1 for the best open 4B comparison model and 64.8 for the best open 8B.
HF Daily Papers' note
The authors trace the gap to caption and question-answer training that rewards retrieving named attributes while leaving motion information unused. They train V-JEPA-LM on transformed video variants whose answers depend on changes in dynamics. At a matched training budget, this supervision adds 4.5 points on average and 38 points on speed questions in natural benchmarks. The authors report that their 4B model averages 72.1 across eleven video benchmarks, against 61.1 for the best open 4B comparison model and 64.8 for the best open 8B.
HF Daily Papers' note
score 5