UniProbe: A Learnable Token-Level Hallucination Detector for Large VLMs using Multi-Structural Internal Representations
UniProbe flags hallucinated LVLM tokens from one frozen-model forward pass, then can resample them during decoding.
The detector builds a directed graph across image patches, query tokens, and generated tokens, using attention weights as relations. It alternates a GNN, ViT, and GRU so relational, visual-spatial, and response-order signals interact. The authors also describe a streaming version for hallucination-aware decoding and a self-adaptation method tied to the LVLM’s own outputs. Across tested LVLM backbones, they report state-of-the-art token-level and object-hallucination detection, with object hallucinations reduced by up to 55% at 1.06x standard-generation latency. HF Daily Papers' note
The detector builds a directed graph across image patches, query tokens, and generated tokens, using attention weights as relations. It alternates a GNN, ViT, and GRU so relational, visual-spatial, and response-order signals interact. The authors also describe a streaming version for hallucination-aware decoding and a self-adaptation method tied to the LVLM’s own outputs. Across tested LVLM backbones, they report state-of-the-art token-level and object-hallucination detection, with object hallucinations reduced by up to 55% at 1.06x standard-generation latency. HF Daily Papers' note
score 4