Visual Grounding Safety in Vision-Language Models
The safety gap is in the output format: models refuse harmful text answers but still return points or boxes.
The paper tests five vision-language models on 15,401 paired harmful requests, changing only whether the model is asked for a text answer or a visual grounding. Refusal rates for grounding lag VQA refusal by 31 to 59 percentage points, and safety prompts do not close the gap. The authors report that fine-tuning Qwen3-VL-8B and VisionReasoner-7B with grounding-form refusals and benign grounding data sharply improves refusal while preserving grounding ability. HF Daily Papers' note
The paper tests five vision-language models on 15,401 paired harmful requests, changing only whether the model is asked for a text answer or a visual grounding. Refusal rates for grounding lag VQA refusal by 31 to 59 percentage points, and safety prompts do not close the gap. The authors report that fine-tuning Qwen3-VL-8B and VisionReasoner-7B with grounding-form refusals and benign grounding data sharply improves refusal while preserving grounding ability. HF Daily Papers' note
score 5