Beyond Visual Similarity: Entity-Aligned Retrieval for Knowledge-Based Visual Question Answering
KBMR replaces visual-match retrieval with entity-aware embeddings for KB-VQA.
The paper says CLIP-style retrievers can miss the right knowledge when the same entity looks different, or when different entities look alike. KBMR uses a multimodal LLM to map images into a semantic space meant to preserve concept identity. It also adds a semantic discriminator to weight entity consistency during training at Wikipedia scale. The authors report gains of up to 14.7% in Recall@1 and 9.4% in end-to-end VQA accuracy over CLIP baselines. HF Daily Papers' note
The paper says CLIP-style retrievers can miss the right knowledge when the same entity looks different, or when different entities look alike. KBMR uses a multimodal LLM to map images into a semantic space meant to preserve concept identity. It also adds a semantic discriminator to weight entity consistency during training at Wikipedia scale. The authors report gains of up to 14.7% in Recall@1 and 9.4% in end-to-end VQA accuracy over CLIP baselines. HF Daily Papers' note
score 4