Say Less, Hallucinate Less? Diagnosing and Restoring Object Coverage in Preference-Optimized Vision-Language Models
Stopping the base model early reproduced 58–86% of the hallucination reduction measured for three preference-optimized checkpoints.
Across four public LLaVA-1.5-7B checkpoints, preference optimization reduced the object categories named without significantly improving set-level object F1 on CHAIR. Forcing longer captions recovered little of the lost coverage; 94–96% of added object mentions repeated categories already named. A training-free method that adds high-confidence categories from the base model brought three checkpoints within one percentage point of base recall on held-out COCO images, while retaining a lower distinct-object hallucination rate than the base.
ArXiv · AI/CL/LG's note
Across four public LLaVA-1.5-7B checkpoints, preference optimization reduced the object categories named without significantly improving set-level object F1 on CHAIR. Forcing longer captions recovered little of the lost coverage; 94–96% of added object mentions repeated categories already named. A training-free method that adds high-confidence categories from the base model brought three checkpoints within one percentage point of base recall on held-out COCO images, while retaining a lower distinct-object hallucination rate than the base.
ArXiv · AI/CL/LG's note
score 5