Kierston & KierstonBench: Simulation of Allentown, PA and Evaluation of Language Models in Crime Investigation and Surveillance Tasks
Most of the 16 models tested made identifications the simulated records did not support.
Kierston generates daily activities and linked telecommunications, financial, administrative and license-plate records for approximately 125,000 simulated Allentown residents. Models search those records through a restricted interface across 32 task families, without access to the underlying events. A few frontier models solved most tasks and withheld conclusions when evidence was insufficient. Even those models sometimes fell for deceptions and assumptions; the authors conclude that none replaced an investigator.
HF Daily Papers' note
Kierston generates daily activities and linked telecommunications, financial, administrative and license-plate records for approximately 125,000 simulated Allentown residents. Models search those records through a restricted interface across 32 task families, without access to the underlying events. A few frontier models solved most tasks and withheld conclusions when evidence was insufficient. Even those models sometimes fell for deceptions and assumptions; the authors conclude that none replaced an investigator.
HF Daily Papers' note
score 5