CORE: Improving Compositional Reasoning in MLLM Embedding via Reranker Distillation
CORE distills a reranker’s compositional judgments into an embedding model to better separate lookalike image-text matches.
The paper targets retrieval failures where models see the same objects but miss different attribute-object bindings. CORE builds candidate lists across five compositional matching levels and trains with a Rank-KL objective to copy the reranker’s ranking. In the authors’ tests, CORE-RERANKER-8B averages 82.7% across COLA, SUGARCREPE++, and NEGBENCH, while CORE-EMBED-8B leads the evaluated embedding models. The gains also transfer to MCMR without hurting COCO and Flickr30K retrieval. HF Daily Papers' note
The paper targets retrieval failures where models see the same objects but miss different attribute-object bindings. CORE builds candidate lists across five compositional matching levels and trains with a Rank-KL objective to copy the reranker’s ranking. In the authors’ tests, CORE-RERANKER-8B averages 82.7% across COLA, SUGARCREPE++, and NEGBENCH, while CORE-EMBED-8B leads the evaluated embedding models. The gains also transfer to MCMR without hurting COCO and Flickr30K retrieval. HF Daily Papers' note
score 5