CaliBench: Are the Stochastic Dynamics of Video World Models Physically Calibrated?
CaliBench finds video world models often miss the actual odds of simple physical outcomes.
The paper tests six image-to-video models across nine scenes where the reference distribution is known, including dice, cards, Galton boards, lottery, and roulette. It separates whether a generation can be scored at all from whether the scored outcomes match the expected distribution. The authors report that models tend to pile probability onto a few outcomes, with most scene-model pairs significantly miscalibrated and Veo 3.1 collapsing to one outcome on dice. Roulette also exposed low scorability when the ball’s final position was ambiguous. ArXiv · AI/CL/LG's note
The paper tests six image-to-video models across nine scenes where the reference distribution is known, including dice, cards, Galton boards, lottery, and roulette. It separates whether a generation can be scored at all from whether the scored outcomes match the expected distribution. The authors report that models tend to pile probability onto a few outcomes, with most scene-model pairs significantly miscalibrated and Veo 3.1 collapsing to one outcome on dice. Roulette also exposed low scorability when the ball’s final position was ambiguous. ArXiv · AI/CL/LG's note
score 5