ModaLens: Measuring Image Sensitivity in Report-Conditioned Medical VLMs
ModaLens tests whether a medical VLM changes its answer when the image is swapped but the report stays the same.
On 3,199 paired MIMIC-CXR cases, MedGemma-27B changed its generated answer in 4.26% of trials when a report was provided, versus 20.94% without the report. The authors read that as report availability reducing image-swap sensitivity under their protocol. They note the labels come from reports, limiting claims about visual correctness, and say the direction replicated in two other model lineages. HF Daily Papers' note
On 3,199 paired MIMIC-CXR cases, MedGemma-27B changed its generated answer in 4.26% of trials when a report was provided, versus 20.94% without the report. The authors read that as report availability reducing image-swap sensitivity under their protocol. They note the labels come from reports, limiting claims about visual correctness, and say the direction replicated in two other model lineages. HF Daily Papers' note
score 4