MCR-Bench Study Finds Leading LLMs' Code-Review Accuracy Collapses as Review Rounds Pile Up
A new benchmark testing seven LLMs on real multi-round GitHub code reviews finds accuracy drops sharply as review rounds increase, exposing weak memory across rounds.
Overview
A new benchmark for evaluating large language models on code review finds that their defect-detection accuracy degrades substantially as a review stretches across more rounds of back-and-forth. The benchmark, called MCR-Bench, tested seven mainstream LLMs on 2,269 real-world multi-round code review tasks pulled from GitHub pull request histories, according to the paper, titled “From Static to Dynamic: Benchmarking Real-World Code Review with MCR-Bench” and posted to arXiv on August 27, 2026.
What We Know
Most prior work on automated code review treats the task as a single-round, static decision, according to the paper: “most approaches oversimplify code review into a single-round, static decision task, which fails to capture the multi-round interactive nature and the complex problem-solving processes inherent in realistic review scenarios.” The researchers built MCR-Bench to close that gap, describing it as “the first defect state-aware benchmark designed for realistic multi-round code review,” according to the paper.
The benchmark covers five programming languages — Python, Java, JavaScript, TypeScript, and C# — and consists of 2,269 real-world multi-round code review tasks, each annotated with fine-grained defect information and cross-round state labels, according to the paper. The tasks were built from real pull request histories: the researchers filtered source repositories down to those with “more than 100 stars,” “sustained commit activity over the past five years,” and “at least 1,500 PRs,” then processed pull requests from those repositories to construct the evaluation set, according to the full paper text. Tasks with three rounds of review are the single largest category, making up 40.86% of the dataset, and the average task contains 2.37 defects, with a maximum of 13 defects in a single task, according to the full paper text.
The researchers evaluated seven models — GPT-5.2, Claude Haiku 4.5, Gemini 3 Flash, DeepSeek-V3.2, Qwen3-Max, GLM-4.7, and Kimi-k2 — on their ability to identify defects and track how those defects evolve across review rounds, according to the full paper text. The headline finding is that accuracy falls as the number of rounds grows: “as the number of review rounds increases, most LLMs exhibit varying degrees of performance degradation, indicating that long-range multi-round interactions substantially increase the difficulty of defect identification and state tracking,” according to the full paper text. For Claude Haiku 4.5, the reported F1 score for defect identification fell from 0.6495 at round two to 0.2857 at round ten — a drop of more than half. GPT-5.2 also declined over the same span, from 0.5995 at round two to 0.5000 at round ten, though less steeply, according to the full paper text.
The paper also breaks down why models fail, sorting errors into named categories. Among false positives, the largest category is “State–Temporal Misalignment,” accounting for 32.5% of false-positive errors, defined as: “Failure to align defect states with code versions, causing already resolved defects to be repeatedly flagged as new,” according to the full paper text. The second-largest false-positive category, “Over-reviewing” at 27.8%, occurs when a model “continues to produce review comments in the absence of real defects, misclassifying suggestions or speculative concerns as defects,” according to the full paper text. On the false-negative side, “Cross-round Defect Forgetting” accounts for 25.1% of errors, where a model “fails to track unresolved defects across review rounds and stops mentioning them prematurely,” followed by “Long-range Dependency Miss” at 23.4% and “Semantic Defect Blindness” at 22.3%, according to the full paper text.
The researchers have released the benchmark’s data-construction pipeline and evaluation code on GitHub, where the project is described as moving “from static to dynamic” code review evaluation “by capturing real-world code review scenarios with multi-round interactions, iterative discussions, and evolving code changes,” according to the GitHub repository. The repository confirms the same five-language coverage as the paper and is released under an Apache-2.0 license, according to the GitHub repository.
What We Don’t Know
The paper does not disclose which specific commercial deployments, if any, currently rely on LLM-based code review at production scale, nor does it claim to test those deployed systems directly — MCR-Bench evaluates the underlying models on a constructed benchmark, not any specific vendor’s shipped code-review product. The paper also does not report results for every model at every individual round beyond the round-two and round-ten comparison points highlighted above, and it is not yet clear whether other research groups have replicated these findings or whether the benchmark data will see broader adoption in industry evaluations.
Analysis
The finding has a direct practical implication for teams already deploying LLM-based code review tools: a model’s accuracy on a single-shot, first-pass review may not predict how well it holds up once a pull request goes through several rounds of comments, fixes, and re-review. The paper’s own stated contributions frame this explicitly as a capability-boundary problem — the authors describe their work as evaluating “the capability boundaries of mainstream LLMs in multi-round code review” and constructing “a fine-grained taxonomy of failure root causes,” according to the full paper text. The two largest failure categories identified — models losing track of which defects were already resolved, and models forgetting about defects they flagged earlier in the conversation — both point to the same underlying weakness: state and memory across a long-running, multi-turn interaction, rather than any single defect-detection error in isolation.