Probing Spatial State Representations in Visual Foundation Models
Reading the visible scene accurately did not guarantee remembering objects out of view or predicting later observations.
The researchers probed ten frozen representations using streams of first-person views of static indoor environments. Results varied by task, network layer and observation-history length. Predicted future features retained recoverable spatial structure, but a spatial reader trained on real features did not transfer unchanged. Predicted actions responded to goal changes, yet supplying the correct goal did not reliably improve progress.
ArXiv · AI/CL/LG's note
The researchers probed ten frozen representations using streams of first-person views of static indoor environments. Results varied by task, network layer and observation-history length. Predicted future features retained recoverable spatial structure, but a spatial reader trained on real features did not transfer unchanged. Predicted actions responded to goal changes, yet supplying the correct goal did not reliably improve progress.
ArXiv · AI/CL/LG's note
score 4