How Can Rhetoric Reward-Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Review
AI reviewers changed scores when the science stayed the same but the rhetoric moved.
The study built 4,200 rewritten manuscripts from 120 anonymized ICLR 2026 submissions and tested five LLM reviewers. Evidence framing and novelty stance had the strongest effects on overall assessments, while scope framing mattered less and other dimensions were weaker or unstable. Score shifts depended on the reviewer’s starting point: low scores tended to rise, high scores tended to fall, and middle scores showed the clearest directional contrasts. Stricter review lowered mean overall assessment by 1.36 points but did not reliably remove rhetorical sensitivity. HF Daily Papers' note
The study built 4,200 rewritten manuscripts from 120 anonymized ICLR 2026 submissions and tested five LLM reviewers. Evidence framing and novelty stance had the strongest effects on overall assessments, while scope framing mattered less and other dimensions were weaker or unstable. Score shifts depended on the reviewer’s starting point: low scores tended to rise, high scores tended to fall, and middle scores showed the clearest directional contrasts. Stricter review lowered mean overall assessment by 1.36 points but did not reliably remove rhetorical sensitivity. HF Daily Papers' note
score 4