How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining
The paper predicts encoder-free multimodal models can match encoder-based ones around 10^22 FLOPs.
The authors compare scaling laws for MLLMs that use pretrained visual encoders against models that learn directly from raw pixels. They find text loss behaves similarly across both designs, while multimodal loss favors encoder-based models at small scale. As compute rises, the encoder-free setup is predicted to close the gap, with the language model increasingly taking on vision-specific processing. HF Daily Papers' note
The authors compare scaling laws for MLLMs that use pretrained visual encoders against models that learn directly from raw pixels. They find text loss behaves similarly across both designs, while multimodal loss favors encoder-based models at small scale. As compute rises, the encoder-free setup is predicted to close the gap, with the language model increasingly taking on vision-specific processing. HF Daily Papers' note
score 5