Grid-Bench: Can LLMs Predict How Discrete Worlds Change?
Across six LLMs, accuracy on the complete final grid drops sharply as the sequence of changes grows longer.
Grid-Bench asks models to predict a final grid from an initial state, explicit rules and a text history of events. It covers eight families of operations across two-, three- and four-dimensional grids, with executable reference results that make every cell checkable. Models that perform similarly on single events diverge substantially over longer histories, and similar exact-match scores can hide large differences in incorrect cell counts. Some models still recover many correct changes over long histories beyond what simply copying the initial grid would achieve.
ArXiv · AI/CL/LG's note
Grid-Bench asks models to predict a final grid from an initial state, explicit rules and a text history of events. It covers eight families of operations across two-, three- and four-dimensional grids, with executable reference results that make every cell checkable. Models that perform similarly on single events diverge substantially over longer histories, and similar exact-match scores can hide large differences in incorrect cell counts. Some models still recover many correct changes over long histories beyond what simply copying the initial grid would achieve.
ArXiv · AI/CL/LG's note
score 4