MNIST-PRO: MNIST is Back as a Partially Observable World for AI Agents
The benchmark turns MNIST into a glimpse-by-glimpse memory test for multimodal agents.
MNIST-PRO makes digit recognition partially observable, forcing agents to search sequentially under lookback constraints. The authors evaluated ten multimodal models across raw visual history, text states, metric grid maps, and a consolidated canvas. Models did well when they could see everything, but partial views exposed failures in integrating fragments, continuing exploration, and revising wrong early guesses. ArXiv · AI/CL/LG's note
MNIST-PRO makes digit recognition partially observable, forcing agents to search sequentially under lookback constraints. The authors evaluated ten multimodal models across raw visual history, text states, metric grid maps, and a consolidated canvas. Models did well when they could see everything, but partial views exposed failures in integrating fragments, continuing exploration, and revising wrong early guesses. ArXiv · AI/CL/LG's note
score 5