When AI Reviews Train AI Reviewers: Scientific-Judgment Collapse and Mitigation
Synthetic peer reviews made later AI reviewers less varied and more compressed in their judgments.
The paper tests a recursive peer-review loop by fine-tuning Llama 3.1 8B on ICLR reviews, then training successors on mixtures of official and model-generated 2024 reviews. Adding synthetic reviews narrowed rating distributions and reduced semantic diversity within papers and across the corpus. The authors call this “scientific-judgment collapse” and propose TrustReviewer, combining curated training data with activation steering to preserve review diversity and improve recommendation alignment. HF Daily Papers' note
The paper tests a recursive peer-review loop by fine-tuning Llama 3.1 8B on ICLR reviews, then training successors on mixtures of official and model-generated 2024 reviews. Adding synthetic reviews narrowed rating distributions and reduced semantic diversity within papers and across the corpus. The authors call this “scientific-judgment collapse” and propose TrustReviewer, combining curated training data with activation steering to preserve review diversity and improve recommendation alignment. HF Daily Papers' note
score 4