How Well Does AI Peer Review Work? A Benchmark Built from 100 Planted Errors
Pooling every system’s output caught 93 of 100 planted errors; the best single system caught 71.
The benchmark tested frontier models and two commercial AI review tools on 10 psychology papers containing deliberately planted errors. Individual systems caught between 30 and 71 errors, with different systems finding different problems. All seven errors missed by every system were omissions—information deleted from the papers. False positives were not measured, and whether the planted errors reflect those in real papers remains unknown.
ArXiv · AI/CL/LG's note
The benchmark tested frontier models and two commercial AI review tools on 10 psychology papers containing deliberately planted errors. Individual systems caught between 30 and 71 errors, with different systems finding different problems. All seven errors missed by every system were omissions—information deleted from the papers. False positives were not measured, and whether the planted errors reflect those in real papers remains unknown.
ArXiv · AI/CL/LG's note
score 4