The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images
The paper argues that crop-and-zoom tool use often looks useful without actually changing the model’s answer.
The authors audit multimodal LLM visual operations with causal interventions at the policy, trajectory, and step level. Across six models and five perception benchmarks, they find marginal or negative gains versus direct inference, with higher token cost. They name two failure modes: “Calling Without Looking,” where observations do not affect the answer, and “Looking Without Planning,” where useful observations are gathered incoherently. Source: ArXiv · AI/CL/LG's note.
The authors audit multimodal LLM visual operations with causal interventions at the policy, trajectory, and step level. Across six models and five perception benchmarks, they find marginal or negative gains versus direct inference, with higher token cost. They name two failure modes: “Calling Without Looking,” where observations do not affect the answer, and “Looking Without Planning,” where useful observations are gathered incoherently. Source: ArXiv · AI/CL/LG's note.
score 5