CogBattery: Human-Referenced Evaluation of Cognitive Task Behavior and Structure in Language Models
Models can match the direction of human cognitive effects while making very different errors.
CogBattery tested 54 open-weight language models on text adaptations of 13 cognitive paradigms, using human results from each source study as references. In n-back tasks, models responded more to lures than controls, as humans do, but made far more false alarms on ordinary non-targets. Most models’ confidence distinguished correct from incorrect answers less well than humans’ did, and most always chose the same wager—even when they classified accurately. Larger models were not consistently closer to human behavior, and predefined cognitive groupings predicted no better than random groupings of the same size.
HF Daily Papers' note
CogBattery tested 54 open-weight language models on text adaptations of 13 cognitive paradigms, using human results from each source study as references. In n-back tasks, models responded more to lures than controls, as humans do, but made far more false alarms on ordinary non-targets. Most models’ confidence distinguished correct from incorrect answers less well than humans’ did, and most always chose the same wager—even when they classified accurately. Larger models were not consistently closer to human behavior, and predefined cognitive groupings predicted no better than random groupings of the same size.
HF Daily Papers' note
score 4