Competence-Based Evaluation of Logical Reasoning in Large Language Models
None of the six GPT and Qwen models met the study’s consistency requirement across the tested ordering problems.
The evaluation tests whether models apply the same reasoning skill as problem size and wording change, rather than reporting a single accuracy score. Models performed reasonably well within bounded complexity ranges but failed to maintain correctness across the full tested family. Chain-of-thought and GRPO extended strong performance without eliminating all tested failures. Supervised LoRA showed little transfer to held-out problems.
HF Daily Papers' note
The evaluation tests whether models apply the same reasoning skill as problem size and wording change, rather than reporting a single accuracy score. Models performed reasonably well within bounded complexity ranges but failed to maintain correctness across the full tested family. Chain-of-thought and GRPO extended strong performance without eliminating all tested failures. Supervised LoRA showed little transfer to held-out problems.
HF Daily Papers' note
score 4