AI Code Review Is a Property of Your Team, Not the Tool

Two teams can run the same AI reviewer on the same class of codebase and report opposite results. The argument that follows is almost always about the tool. It is the wrong argument. More than one in five code reviews on GitHub now involve an agent (The GitHub Blog: Agent pull requests are everywhere¹). The interesting variable is no longer whether agents are in your review process (they are), but what your team does with what they produce.

Both Verdicts on AI Code Review Are Universal Claims

The two loudest positions on AI and code review are both stated as facts about the industry, and neither one is a fact about your team. One camp says review is broken. The other says review is fine if you follow the checklist. Both skip the step where you find out which one describes you.

The “AI broke review” pole is best stated by Moderne, writing up an interview with Morgan Stanley engineers:

“In the time that Dov and I have been speaking, you probably could have produced a thousand-file, 10,000-line PR on the back of just a simple prompt. No human here is going to review that.”

That is a real observation from a real org, and Moderne sells the deterministic tooling it proposes as the fix. The prescription pole reads the same way from the other direction. GitHub’s guidance hands every reader the same review habits regardless of who is reviewing, and CIO’s exec-facing version diagnoses a “crisis” and prescribes restructuring. CIO’s author earns one credit: he names a real metric pair, writing that he only expands scope “when acceptance rate is climbing and the post-merge defect rate is holding flat or falling. One signal moving without the other is a red flag” (CIO: The code review crisis and how you should rebuild review models³).

The closest thing to a contingent position is Addy Osmani’s, and he gets the shape of the answer right:

“The only answer that survives contact with a real codebase is that it depends entirely on who you are.”

His three variables are blast radius, how long the code lives, and how many people need to understand it (Addy Osmani: Agentic Code Review⁴). Those are properties of the code. They tell a solo developer to relax and an enterprise to be careful, which is useful. They say nothing about whether the specific humans on your review rotation can catch what an agent gets wrong. A CMU team working from 3,100 coded practitioner documents puts the missing variable where it belongs: “review is the control point through which a coding agent’s effect on software is decided, and that AI does not fix the sign of that effect: the team sets it, through the expertise its humans bring and how it structures the review process” (Agarwal et al.: 3100 Opinions on Code Review in an AI World⁵). That paper is an unrefereed preprint and says so plainly: its authors call it “a proposed explanatory theory, not a validated one.” Treat it as a set of falsifiable propositions, not a result.

Both camps cite the same number to defend their position: the share of agent-authored PRs at your company that received independent human review. Call it agent_pr_review_rate. It is on every engineering dashboard that tracks agents, both poles wave at it, and neither one asks what it counts. The rest of this post takes that one number apart, because everything the discourse is arguing about is downstream of its definition.

Find Out Whether Your Metric Counts the Invoking Developer

Before you act on your agent_pr_review_rate, go find the query behind it and answer one question: when a developer runs an agent and then approves the PR the agent opened, does your dashboard count that as an independent review? The direction of your entire trend line depends on the answer, and most teams have never chosen one.

The CMU paper hit this inside its own dataset, and disclosed it:

“Whether the developer who invoked an agent counts as an independent reviewer or as the author reviewing their own work decides whether independent review of agent PRs sits above or below the human rate.”

The underlying number is 40.1% of agent PRs examined only by the developer who invoked the agent, versus 21.5% of human PRs (Agarwal et al.⁵). Here is what that means in a query:

-- Definition A: any human approval counts.
--   The developer who invoked the agent is a reviewer.
reviewed := EXISTS (SELECT 1 FROM reviews r
                    WHERE r.pr_id = pr.id
                      AND r.author_type = 'human')

-- Definition B: a review counts only if the reviewer is not the invoker.
reviewed := EXISTS (SELECT 1 FROM reviews r
                    WHERE r.pr_id = pr.id
                      AND r.author_type = 'human'
                      AND r.author_id <> pr.agent_invoked_by)

Under Definition A, that 40.1% slice is reviewed code and agent PRs look better supervised than human PRs. Under Definition B, the same slice is self-review and agent PRs look worse. Same data, same quarter, opposite headline. No engineer wrote a bug. Be precise about the scope of this: only the independent-review construct reverses. Merge speed and review-discussion volume are reported as stable in the same paper, so “all the agent-PR trends flip” overstates it.

Salesforce Engineering published the worked case of what happens when nobody asks. Their write-up reports code volume up roughly 30% with PRs regularly past 20 files and 1,000 lines, and then interprets a metric that had stopped rising and in places was falling outright:

“More concerning, review time for the largest pull requests began to plateau […] This indicated that reviewers were no longer meaningfully engaging with changes.”

Declining review time on large PRs is genuinely ambiguous. It is consistent with disengagement, and it is equally consistent with better tooling, better PR descriptions, or previously-inflated baselines. The post entertains no alternative, discloses no population or sample size, and (the part that matters here) never says who counts as a reviewer anywhere in it. That is not sloppiness on their part so much as the industry default. It is a footgun: a metric reported without its numerator defined will be read as whatever the reader already believes.

There is a formal name for the defect. An RCT-methods group working on AI evaluation puts it this way: “When construct validity is weak, a study may show that scores increased with AI, but it cannot credibly claim that human capability in the targeted domain has improved” (Kelly et al.: Principles and Guidelines for RCTs in AI Evaluation⁷). Their two named failure modes, construct underrepresentation and construct-irrelevant variance, describe an agent_pr_review_rate exactly. The first question is therefore not whether AI is hurting your review process. It is which of those two SQL predicates your dashboard is running.

Code Review Already Had a Finding Dissolve This Way

The claim “a tool arrived and review discussion went down” is not new. The last time the field made it about code review at scale, the finding did not survive its own analysis choices. The 2026 claim has the same shape as the 2020 one.

In 2020, a SANER paper studied 685 GitHub projects and found that “with the introduction of CI, pull requests are being discussed less. On average CI saves up to one review comment per pull request” (Cassee, Vasilescu, Serebrenik: The Silent Helper⁸). Reasonable study, reasonable finding, exactly the sort of thing a VP would put on a slide.

In 2026, a TOSEM-accepted multiverse analysis re-ran it. The authors identified nine pivotal analytical decisions in the original, each with at least one equally defensible alternative. They “systematically reran all the 3,072 resulting analysis pipelines on the original dataset” (Cassee & Feldt: Exploring the Garden of Forking Paths in Empirical Software Engineering Research⁹). The result:

“Interestingly, only 6 of these universes (<0.2%) reproduced the published results; the overwhelming majority produced qualitatively different, and sometimes even opposite, findings.”

One decision did most of the damage: “For the full study, only changing the Period Length resulted in a different outcome in 86.3% of universes” (Cassee & Feldt⁹). The multiverse paper names that 2020 study as its target and discloses that “one of the authors of this study was also involved in the primary study” (Cassee & Feldt⁹).

The last row is the one to act on. Nobody has re-run your dashboard 3,072 ways, so you do not know which cell of that multiverse your quarterly number came from. This is the same discipline I applied to model selection in the leaderboard-noise post. The number you generate from your own workload beats the number you import, but only if you know which choices produced it.

Two honest limits. First, this style of analysis has a live critique. Del Giudice and Gangestad argue that “if specifications are not truly arbitrary, multiverse-style analyses can produce misleading results, potentially hiding meaningful effects within a mass of poorly justified alternatives” (Del Giudice & Gangestad: A Traveler’s Guide to the Multiverse¹⁰). Instability is evidence that a number needs its choices justified, not proof that the effect is imaginary. Second, the method’s originators are careful about what it does. They hope it “raises awareness that, in the light of the multiverse of statistical results, isolating a single statistical result stemming from a chain of arbitrary choices can be highly misleading,” and say plainly that it “is not a formal test of questionable research practices” (Steegen et al.: Increasing Transparency Through a Multiverse Analysis¹¹).

The most credible version of this move is first-party. METR redesigned its developer-productivity experiment after finding that developers were opting out rather than work without AI. It wrote against its own headline number: “However the true speedup could be much higher among the developers and tasks which are selected out of the experiment” (METR: We are Changing our Developer Productivity Experiment Design¹²). It also bound the damage: “The selection effects seem to affect a minority share of developers and of tasks, which limits the degree of bias” (METR¹²). Bounding the bias like that is what a real correction looks like. That is a productivity study, not a review study. Take the transfer for what it is: a demonstration that a headline effect can be partly a property of who got measured.

I ran the same failure inside this pipeline. My research pass marked every figure in the Faros AI telemetry report as refuted; a dedicated re-check found all of them present verbatim and server-rendered on Faros’s own domain. The refutation was a fetch failure scored as an absence. Nobody fabricated anything, one automated retrieval step failed quietly, and a verified source came within one edit of being called wrong in public.

Nobody Has to Be P-Hacking for This to Happen

Stop treating “we only ran the analysis once” as a defense; it is the exact case the statistics literature was written about. Gelman and Loken state it in their abstract: “Researcher degrees of freedom can lead to a multiple comparisons problem, even in settings where researchers perform only a single analysis on their data” (Gelman & Loken: The garden of forking paths¹³).

The mechanism is the part engineers underrate. They write that “researchers can perform a reasonable analysis given their assumptions and their data, but had the data turned out differently, they could have done other analyses that were just as reasonable in those circumstances.” Those choices, they add, “do not feel like degrees of freedom because, conditional on the data, each choice appears to be deterministic” (Gelman & Loken¹³). Your data engineer picked one reviewer-identity predicate. Conditional on the data in front of them, it felt like the only sensible one. It was one path through the garden, and the fork was never logged.

A Bigger Dashboard Does Not Fix It

When an agent-PR metric looks ambiguous, the reflex is to add more data: longer windows, more repos, more teams. That reflex makes the problem worse in a specific, measurable way. Furia and Torkar note that sampling more datapoints “may just entrench our reliance on the biased estimate by reducing its variance and giving the false impression of reliability or ‘significance’” (Furia & Torkar: Mitigating Omitted Variable Bias in Empirical Software Engineering¹⁴).

Their SE case study shows how little confounding it takes: they compute that a modest omitted-variable strength “would be sufficient to flip the sign of the measured effect” (Furia & Torkar¹⁴). That is a sensitivity threshold, not an observed reversal. The move that actually helps is not more rows, it is conditioning on the variable you left out. For agent_pr_review_rate, the omitted variables are the three moderators below.

Moderator One: Measure Verification Capacity, Not Stated Skepticism

Author’s judgment. The evidence below measures individual verification behavior and individual reviewer outcomes; extending it to “your team’s review rotation” is my inference, not a claim any of these papers makes. Fawzy et al. studied solo verification by Prolific-recruited individuals; the phrase “code review” appears twice in the entire paper. The premises are theirs; the transfer to team review is mine.

Your team’s survey score on “do you trust AI-generated code” tells you nothing about whether they can catch its errors. Awareness of the risk is broadly distributed. The ability to act on it is not.

That is the finding, near-verbatim, from a study of 162 vibe coders across three experience groups: a general awareness of risks in AI-generated code is broadly distributed, but “the capacity to evaluate, debug, and verify remains experience-dependent” (Fawzy, Tahir, Blincoe: From Prompting to Verification¹⁵). Reported perceptions of code quality came out broadly similar across groups while actual practices diverged sharply. Professionals reported the highest rates of consistent checking, “with approximately 45% reporting that they always check” AI-generated code before using it; non-developers were the only group to report never checking at all (Fawzy et al.¹⁵).

Measured review outcomes point the same way. Mining 28,127 Mozilla reviews, Kononenko et al. found “that overall 54% of Mozilla code reviews missed bugs in the approved commits,” and that “less experienced developers […] are more likely to neglect problems in changes under review,” where less experienced means having conducted relatively fewer code review tasks (Kononenko et al.: Investigating Code Review Quality¹⁶). Their companion survey of 88 Mozilla developers found the same engineers naming familiarity as a driver of quality: “the review quality is primarily associated with the thoroughness of the feedback, the reviewer’s familiarity with the code, and the perceived quality of the code itself” (Kononenko, Baysal, Godfrey: Code Review Quality: How Developers See It¹⁷). That is perception, not a measured outcome. Worth labeling as such even when it agrees.

The same-tool-opposite-outcome case exists inside a single company. In interviews at BNY Mellon, researchers recorded that “peer review of code is critical to deploying reliable software to production. However, the use of AI assistants in producing such code may have unforeseen impacts when it comes to the review process,” including a senior manager’s recurring issue with junior developers optimizing pieces of a change that were not the right things to optimize (Chen et al.: Beyond the Commit¹⁸). One organization, one assistant, different results by seniority.

Every one of those signals is about the human sitting in the numerator of agent_pr_review_rate. A review by someone who cannot evaluate the change increments the metric identically to a review by someone who can. The number cannot tell them apart, which is why the per-diff framework in evaluating AI coding agent output helps in direct proportion to who is holding it.

Moderator Two: Measure the Bot on Your Own Repo

Author’s judgment. The imperative to measure your own bot is mine, inferred from a limitation the source discloses rather than from a finding it reports: Zhong et al. pool their results across 300 projects with no breakdown by project, team, or reviewer, and flag that their conclusions “may not generalize to proprietary enterprise systems or smaller niche repositories.” Adapting reliance-calibration metrics from human-AI decision research to code review is likewise my move; that literature contains no software engineering content at all.

Compute your own reviewer bot’s suggestion-adoption rate and false-positive rate before you let it absorb review load. The published averages are pooled across projects that are not yours. They are still worth knowing, and they are not flattering.

Across 278,790 code review conversations in 300 open-source GitHub projects, human reviewers achieved “significantly higher adoption rates (56.5% vs. 16.6%)” than AI agents, and:

“Over half of unadopted suggestions from AI agents are either incorrect or addressed through alternative fixes by developers.”

When the suggestions are taken, they cost something: adopted agent suggestions “produce significantly larger increases in code complexity and code size than suggestions provided by human reviewers” (Zhong et al.¹⁹). And the failure is quiet on the other side too. An MSR 2026 study of 3,858 PRs found average max redundancy of 0.2867 for AI agents against 0.1532 for humans (roughly 1.87x, significant at p<0.001), while reviewer sentiment ran more positive toward the AI-generated code (Huang et al.: More Code, Less Reuse²⁰). Their reading:

“This disconnect suggests that the surface-level plausibility of AI code masks redundancy, leading to the silent accumulation of technical debt in real-world development environments.”

Note the measurement caveat before you quote that at your team: the sentiment signal comes from an off-the-shelf, general-English classifier. The authors say it “may misinterpret technical discussions in code reviews as neutral or negative emotions” (Huang et al.²⁰).

Run these five steps against your own repo. Each produces a number your dashboard does not currently have:

1. Pull every comment your bot left on merged PRs in the last 90 days. That file is your denominator.

   gh pr list --repo <owner>/<repo> --state merged --limit 500 \
     --json number,mergedAt \
     --jq '.[] | select(.mergedAt > "2026-05-20") | .number' \
   | while read -r pr; do
       gh api "repos/<owner>/<repo>/pulls/$pr/comments" \
         --jq ".[] | select(.user.login == \"<your-bot>[bot]\") | {pr: $pr, path: .path, id: .id}"
     done > bot-comments.jsonl

2. Sample 50 unadopted comments and classify each one as correct-but-ignored, incorrect, or superseded by a different fix. The pooled baseline says over half of unadopted suggestions land in the last two buckets; find out whether yours do.

3. Diff the complexity delta on adopted suggestions. If your bot’s accepted comments reliably grow the change, it is buying you review coverage with maintenance cost.

4. Score reliance, not trust. Count the times a human rejected a wrong bot comment and got it right independently, and the times a human accepted a correct bot comment they had initially missed. Those two counts are the code-review analogue of RSR and RAIR, objective reliance metrics from a literature that finds “trust measurements do not inform users’ appropriate reliance on AI systems” (Raees & Papangelis: From Trust to Appropriate Reliance²¹).

5. Decide, in writing, whether a bot comment increments agent_pr_review_rate. Not whether it should in principle; whether it currently does in your query.

Step five is the one that changes the number. If a bot approval counts, your independent-review rate is measuring bounded, verifiable machine attention and calling it human oversight. If it does not count, you may be under-reporting real coverage. Either answer is defensible. Not having chosen is the only indefensible option, because it means your delegation policy was set by whoever wrote the query.

Moderator Three: Check Whether Your Review Policy Actually Changed

Go open your written review policy right now. If it predates agents opening PRs, you are running the old process on a new kind of input and calling the output a measurement.

The CMU theory names this as a first-class construct: a review governance policy “sets what must be reviewed, by whom, and against what checklist” (Agarwal et al.⁵). Those three clauses are exactly the ones agent PRs invalidate. “What must be reviewed” assumed a human decided the change was worth making. “By whom” assumed the author was not also the invoker. “Against what checklist” assumed the failure modes were human failure modes (typos, off-by-ones, missed edge cases) rather than plausible-looking redundancy that reads clean. The same paper’s proposition P15 states the cost of leaving it stale: “Low review depth and opacity increase comprehension debt, undermining review skill, maintainability, collective ownership, and knowledge transfer” (Agarwal et al.⁵).

A 2026 vision paper argues the reviewer’s job description has already changed, describing a world in which “reviewers transition from manual inspectors into supervisory operators of agents” (Kamalı, Tuna, Haratian, Tüzün: Rethinking Code Review in the Age of AI²²). It is a position paper with no new data, so read it as a proposal rather than a finding. Its concrete criteria are usable: “the selected PR reviewers should possess familiarity with the code changes, have adequate experience in code review, and not be burdened with excessive workload,” and “code should not be reviewed faster than 200 lines per hour to maintain inspection quality” (Kamalı et al.²²). Hold a 20-file, 1,000-line agent PR against 200 lines per hour: that is a five-hour review, a capacity question before it is a quality question. The reviewer-hours ceiling post sizes that side of it.

Audit the policy against these six items. Verb-first, all answerable today:

1. Grep the document for “agent,” “assistant,” and the name of the tool your team actually uses. Zero hits means the policy predates the input it governs.
2. Find the sentence that defines who may approve. Check whether it excludes the person who invoked the agent. If it is silent, your dashboard’s SQL is your policy.
3. Read the checklist and mark every item a compiler or linter already enforces. Those items are dead weight on an agent PR; the surviving items are your real review.
4. Add one item the old checklist could not have had: duplicated logic that already exists elsewhere in the repo, since that is the specific defect class agent PRs over-produce.
5. Write down a size threshold above which a PR must be split or reviewed by two people, and pick the number from your own review-throughput data rather than from a blog post.
6. Name an owner for the policy and a date it gets re-read. An unowned policy drifts back to the dashboard’s default within a quarter.

Item two is the running thread again, one layer down. Most teams have no policy sentence defining independent review at all. The numerator of agent_pr_review_rate was never chosen by anyone accountable for it; it was inherited from a query. A metric definition that nobody chose is still a policy. It is just a policy with no author.

The Strongest Case Against All of This

The best available counter-evidence says team quality buys you nothing, and it deserves a real hearing rather than a footnote. Faros AI analyzed two years of telemetry across 22,000 developers and more than 4,000 teams. It compared each organization against itself between its periods of lowest and highest AI adoption. Their finding is blunt:

“High-performing engineering organizations, those with mature DevOps practices, high DORA metrics scores, and disciplined delivery processes, are experiencing the same downstream deterioration as everyone else.”

The review numbers underneath it are large (Faros AI²³):

Osmani reaches the same conclusion from the same data, writing that “the detail I keep returning to is that teams with mature, disciplined engineering practices were hit just as hard as everyone else” (Addy Osmani: Agentic Code Review⁴). If maturity is not a shield, a diagnostic that sorts teams by maturity is theater.

That is a direct contradiction of DORA’s position, and Faros says so by name. DORA’s 2025 report frames it the other way: “AI’s primary role is as an amplifier, magnifying an organization’s existing strengths and weaknesses. The greatest returns on AI investment come not from the tools themselves, but from a strategic focus on the underlying organizational system” (DORA: State of AI-assisted Software Development 2025²⁴). Weight that appropriately. DORA is a program run by Google Cloud, the report is presented by Google Cloud, and its headline numbers are self-reported perception rather than telemetry. The amplifier thesis as stated is close to unfalsifiable: any outcome confirms it. I would not lean on it to win this argument.

So take Faros seriously, and then check what it measured. Two things, in their own framing. First, the scope: these figures track “metric change between periods of lowest and highest AI adoption within each organization” (Faros AI²³). That is all PRs at high-adoption orgs, not agent-authored PRs specifically. Reading it as “agent PRs get worse review” is precisely the construct swap this post has spent three sections arguing against. I am not going to commit it in the direction that helps me. Second, the window: two years spanning multiple model generations with no stated control for which model wrote the code. Neither point disputes a single percentage, and I have not found anyone who does.

Here is where it leaves the argument, honestly. Faros shows that maturity, measured as DORA scores and delivery discipline, did not protect orgs from an aggregate deterioration. Nothing in it measures the three moderators. Nobody has published a study that stratifies agent-PR review outcomes by reviewer expertise distribution, bot capability on the specific repo, or policy adaptation. The claim is not that a good team is safe. It is that the variables that would decide it are unmeasured, and you are the only person who can measure them on your team.

Run the Diagnostic on Your Own Team

Author’s judgment. The six questions below are my synthesis, not a validated instrument. They follow from the CMU paper’s three proposed moderators, the construct-validity failure in the agent_pr_review_rate definition, and the multiverse result on analytical instability. But no source packages them as a diagnostic, and the paper closest to doing so calls its own theory unvalidated.

Answer these before you accept any published verdict about AI and code review, then re-read your own number under the answers.

Work down the rows and you end up with a different number than the dashboard’s, plus a written record of every choice that produced it. That record is the deliverable. The number by itself is a claim about your team that nobody on your team made.

Then take the second question, the one you can answer this afternoon: pull the query, find the reviewer-identity predicate, and write down which one it uses. If your agent_pr_review_rate is trending in a direction you have opinions about and the predicate was never chosen deliberately, you do not yet have a trend. You have a definition. Once you know which direction AI is moving your review quality, the next question is how much review capacity you have to spend on it. The reviewer-hours ceiling is where that number lives.