DecoupleMix: Decoupled Ratio Search and Convex Allocation for Scalable VLM Data Recipes
DecoupleMix turns VLM pretraining data selection into an optimization problem instead of a hand-tuned recipe.
The paper splits mixture design into inter-class capability ratios and intra-class dataset allocation. It uses iterative search for the first part, then scores datasets for quality and difficulty before selecting them through constrained convex optimization. The authors say small-scale ratio searches transfer to larger runs without retuning, and report stronger results than heuristic baselines. Their VLM, trained with 80B additional multimodal continue-pretraining tokens, is described as competitive with open-source models using larger multimodal budgets. HF Daily Papers' note
The paper splits mixture design into inter-class capability ratios and intra-class dataset allocation. It uses iterative search for the first part, then scores datasets for quality and difficulty before selecting them through constrained convex optimization. The authors say small-scale ratio searches transfer to larger runs without retuning, and report stronger results than heuristic baselines. Their VLM, trained with 80B additional multimodal continue-pretraining tokens, is described as competitive with open-source models using larger multimodal budgets. HF Daily Papers' note
score 5