Same Formulas, Different Semantics: Do Language Models Follow Modal Logic Specifications?
Identical modal formulas flipped labels when the stated semantics changed.
Andrieu and Sileo built paired modal-logic problems where the premises and conjecture stay the same, but frame or domain conditions make the answer reverse. Automated reasoning supplied the labels, with a balanced core meant to stop models from guessing from the condition alone. In direct prompting, four of five recent models scored below that condition-only baseline on the core set. DeepSeek V4 Flash jumped from 4.4% to 88.1% when reasoning mode was enabled on the same prompts. ArXiv · AI/CL/LG's note
Andrieu and Sileo built paired modal-logic problems where the premises and conjecture stay the same, but frame or domain conditions make the answer reverse. Automated reasoning supplied the labels, with a balanced core meant to stop models from guessing from the condition alone. In direct prompting, four of five recent models scored below that condition-only baseline on the core set. DeepSeek V4 Flash jumped from 4.4% to 88.1% when reasoning mode was enabled on the same prompts. ArXiv · AI/CL/LG's note
score 5