MNIST-PRO: MNIST is Back as a Partially Observable World for AI Agents
Partial visibility, not digit recognition, is where the models broke.
MNIST-PRO turns MNIST into a sequential glimpse task so agents have to search, remember, and revise what they think they saw. The authors tested ten multimodal models with four memory formats, from raw visual history to structured grid maps and a consolidated canvas. Full observability was easy; partial observability exposed failures in integrating fragments, continuing exploration, and updating early wrong guesses. HF Daily Papers' note
MNIST-PRO turns MNIST into a sequential glimpse task so agents have to search, remember, and revise what they think they saw. The authors tested ten multimodal models with four memory formats, from raw visual history to structured grid maps and a consolidated canvas. Full observability was easy; partial observability exposed failures in integrating fragments, continuing exploration, and updating early wrong guesses. HF Daily Papers' note
score 4