From Tokens to Semantics: Leveraging Complementary Signals for Hallucination Detection in Black-Box LLMs
A hybrid detector using both semantic disagreement and token-level uncertainty outperformed single-signal approaches only part of the time.
The paper tests hallucination detection when only black-box API outputs are available and no trusted reference text can be used. It compares semantic entropy, token-logprob uncertainty, a TopK token aggregation method, CoCoA, and two supervised methods called Gated and Stacked. Across seven benchmarks and four models, Stacked was best in nearly half the cases, while TopK and CoCoA stayed competitive without labels but needed careful threshold calibration. The authors conclude that no method is universally strongest, especially under low false-positive-rate budgets.
Source: ArXiv · AI/CL/LG's note
The paper tests hallucination detection when only black-box API outputs are available and no trusted reference text can be used. It compares semantic entropy, token-logprob uncertainty, a TopK token aggregation method, CoCoA, and two supervised methods called Gated and Stacked. Across seven benchmarks and four models, Stacked was best in nearly half the cases, while TopK and CoCoA stayed competitive without labels but needed careful threshold calibration. The authors conclude that no method is universally strongest, especially under low false-positive-rate budgets.
Source: ArXiv · AI/CL/LG's note
score 4