From Static to Dynamic: Benchmarking Real-World Code Review with MCR-Bench
MCR-Bench tests whether LLMs can follow defects through multi-round code review, not just flag them once.
The paper introduces a benchmark of 2,269 real-world review tasks across five programming languages, annotated with defect metadata and state changes across rounds. Its experiments find mainstream LLMs struggle with both defect detection and tracking defect lifecycles, especially as interactions get longer. The authors say misses are more common for semantically complex or low-salience defects, and trace failures to issues such as temporal misalignment across rounds and weak long-range memory. ArXiv · AI/CL/LG's note
The paper introduces a benchmark of 2,269 real-world review tasks across five programming languages, annotated with defect metadata and state changes across rounds. Its experiments find mainstream LLMs struggle with both defect detection and tracking defect lifecycles, especially as interactions get longer. The authors say misses are more common for semantically complex or low-salience defects, and trace failures to issues such as temporal misalignment across rounds and weak long-range memory. ArXiv · AI/CL/LG's note
score 5