A landmark study of 1,689 Copilot-generated programs found 40% contained vulnerabilities. Every one of those PRs almost certainly had a human’s name on the approval.
That’s the part of “human in the loop” nobody likes to examine too closely. A name on a PR isn’t proof that judgment happened. It’s proof someone clicked approve.
I’ve sat on both sides of that approval button enough times to know how easy it is to convince yourself you reviewed something when what you actually did was skim it.
What AI code review is genuinely good at
Worth saying plainly: automated review tools catch real problems, fast, at a scale no human review queue can match. Pattern-matching against known vulnerability classes, style consistency, common bug signatures – this is exactly the kind of deterministic, high-volume work AI does better than a tired reviewer on their fourth PR of the afternoon. Teams that use it this way are right to.
The trouble starts when that same speed gets mistaken for a substitute for the review a regulated product still needs.
Where “reviewed” quietly stops meaning anything
SmartBear’s analysis of a Cisco Systems team found defect detection degrades sharply past roughly 400 lines in a single review. A single AI-assisted feature can blow past that in one prompt, arriving as 600 or more lines that a developer skims, watches the tests pass, and merges.
That’s not a hypothetical failure mode. It has a name: automation bias, the well-documented tendency to over-trust automated output and stop scrutinizing it. A 2012 systematic review found erroneous automated advice was followed 26% more often among groups relying on automated recommendations, and the pattern shows up just as reliably with code. Reviewing AI-generated code takes real, measurable extra cognitive effort – 38% of developers say so – which is exactly the condition under which people default to checking whether tests pass and moving on.
My honest read is that AI didn’t create this problem so much as expose it at a volume nobody could keep ignoring. The review process strong enough to actually catch what mattered was usually one dedicated engineer quietly going above and beyond, not something the team as a whole could reliably count on. AI just generates code faster than any one person’s extra effort can keep covering for that gap.
Why a named reviewer isn’t the same as accountability
Regulated code needs a named human owner, not a rotating committee. That part isn’t really controversial. What’s less obvious is the gap underneath it: a name on a PR can exist without the judgment that name is supposed to represent ever really happening.
Call it borrowed accountability. The signature is real. The scrutiny behind it wasn’t. And the categories where that gap is most dangerous in a regulated product are specific, not general:
Whether code matches regulatory intent, not just whether it runs. AI can produce code that’s syntactically correct and functionally passes tests while missing what a compliance requirement actually meant. Catching that requires someone who knows the requirement, not just the codebase.
Architectural decisions with compliance consequences downstream. A change to how data is logged or retained can look like a minor refactor and still be the difference between an auditable system and one that isn’t, months later.
The judgment calls on what needs escalation. AI can flag anomalies. It can’t decide which anomaly is the one that matters in a specific regulatory context, because that decision requires context AI wasn’t given and often can’t be given.
None of these are things AI is incapable of processing. They’re things where the accountability itself has to sit with a person, because “the model reviewed it” is not an answer that survives being asked twice.
What this looks like in practice
The fix isn’t more review. It’s review sized so scrutiny is genuinely possible, aimed at the categories that actually need it:
- Cap what goes into a single review at the size a human can realistically hold in their head – closer to that 400-line ceiling than a 900-line end-of-sprint diff.
- Route the three categories above to a named reviewer explicitly, every time, rather than folding them into general review and hoping they get caught.
- Before approving, require the reviewer to be able to state what “wrong” would have looked like. If they can’t articulate the failure mode, they’re rubber-stamping, not reviewing.
None of this is expensive to put in place. It’s mostly a willingness to admit that “reviewed” has been doing a lot of unearned work in most teams’ process, and to stop letting a name on a PR stand in for the judgment it’s supposed to represent.
The next time someone tells you a PR was reviewed, what would you actually ask to find out if that’s true?
If you want to talk through this more, feel free to reach out on LinkedIn.