Unbundling Multimodal Vision Encoding
Dedicated vision capacity mattered in these tests; a pretrained vision encoder did not.
The researchers compared models using common training data and matched total compute to isolate three vision-encoder design choices. An encoder trained jointly from scratch matched a pretrained one, while removing dedicated vision capacity hurt visual understanding even when the language backbone gained the equivalent parameters. Vision-only modules placed inside early backbone layers performed comparably to a separate encoder and could reduce time to first token. The findings also held across backbone sizes and when starting from a pretrained language model.
HF Daily Papers' note
The researchers compared models using common training data and matched total compute to isolate three vision-encoder design choices. An encoder trained jointly from scratch matched a pretrained one, while removing dedicated vision capacity hurt visual understanding even when the language backbone gained the equivalent parameters. Vision-only modules placed inside early backbone layers performed comparably to a separate encoder and could reduce time to first token. The findings also held across backbone sizes and when starting from a pretrained language model.
HF Daily Papers' note
score 5