Visual Contrastive Self-Distillation
VCSD turns image-content removal into the teacher signal, without an external teacher or privileged supervision.
The method compares an EMA teacher’s next-token distributions with the original image versus a content-erased control, then sharpens the original-image target where visual content raises likelihood. The student distills that full distribution during on-policy generation. On ViRL39K, it beats matched OPSD across Qwen3-VL and Qwen3.5 models, including a Qwen3-VL aggregate gain from 62.27% to 67.04% at 2B. The paper says it adds no inference-time cost and needs no reasoning traces, visual evidence signals, or privileged answers. HF Daily Papers' note
The method compares an EMA teacher’s next-token distributions with the original image versus a content-erased control, then sharpens the original-image target where visual content raises likelihood. The student distills that full distribution during on-policy generation. On ViRL39K, it beats matched OPSD across Qwen3-VL and Qwen3.5 models, including a Qwen3-VL aggregate gain from 62.27% to 67.04% at 2B. The paper says it adds no inference-time cost and needs no reasoning traces, visual evidence signals, or privileged answers. HF Daily Papers' note
score 5