Say the Same, Act Differently: Diagnosing and Steering Text-Action Divergence in Reasoning VLAs
Visual attacks triggered new collisions in 52.6% of tested driving scenes while leaving the model’s rationale word-for-word unchanged.
The researchers tested 154 obstacle-annotated scenes in the Alpamayo driving model; restricting perturbations to a localized forward-camera patch reduced that rate to 35.1%. Their TNAS method identifies directions that strongly affect actions while barely affecting generated text, outperforming unguided action attacks, which produced the same collision-and-unchanged-rationale outcome in 4.55% of scenes. Tests on the InstructVLA manipulation model also shifted actions while preserving exact responses on 85.0% of examples. The authors conclude that safety monitors must check predicted actions directly, because unchanged explanations cannot certify safe behavior.
HF Daily Papers' note
The researchers tested 154 obstacle-annotated scenes in the Alpamayo driving model; restricting perturbations to a localized forward-camera patch reduced that rate to 35.1%. Their TNAS method identifies directions that strongly affect actions while barely affecting generated text, outperforming unguided action attacks, which produced the same collision-and-unchanged-rationale outcome in 4.55% of scenes. Tests on the InstructVLA manipulation model also shifted actions while preserving exact responses on 85.0% of examples. The authors conclude that safety monitors must check predicted actions directly, because unchanged explanations cannot certify safe behavior.
HF Daily Papers' note
score 5