Unbundling Multimodal Vision Encoding
Dedicated vision capacity mattered in these tests; separate vision pretraining did not.
The researchers compared models trained on common data at matched total training compute, testing visual question answering and language performance. A vision encoder trained jointly from scratch matched a pretrained encoder. Removing the encoder hurt visual understanding even when its parameter budget was added to the language backbone. Vision-only modules placed within early backbone layers performed comparably to a separate encoder and could reduce time to first token.
ArXiv · AI/CL/LG's note
The researchers compared models trained on common data at matched total training compute, testing visual question answering and language performance. A vision encoder trained jointly from scratch matched a pretrained encoder. Removing the encoder hurt visual understanding even when its parameter budget was added to the language backbone. Vision-only modules placed within early backbone layers performed comparably to a separate encoder and could reduce time to first token.
ArXiv · AI/CL/LG's note
score 5