PHRBench: A Behavioral Evaluation of Post-Hallucination Reasoning in LLMs
PHRBench tests whether models can recover after a hallucinated premise enters their reasoning context.
The paper introduces a controlled benchmark across four domains and 18 large language models. It scores reasoning paths by hallucination compliance, hallucination avoidance, and heuristic correction, separately from final-answer correctness. In 4,820 instances, successful recovery was relatively rare and correlated with more frequent belief updates during reasoning. The authors also report that features of the hallucinated prompt predicted recovery, with a lightweight predictor reaching 0.847 AUROC. ArXiv · AI/CL/LG's note
The paper introduces a controlled benchmark across four domains and 18 large language models. It scores reasoning paths by hallucination compliance, hallucination avoidance, and heuristic correction, separately from final-answer correctness. In 4,820 instances, successful recovery was relatively rare and correlated with more frequent belief updates during reasoning. The authors also report that features of the hallucinated prompt predicted recovery, with a lightweight predictor reaching 0.847 AUROC. ArXiv · AI/CL/LG's note
score 5