Reviewing Code You Didn't Write

Ask engineers why they review each other's code and most will say: to catch bugs. When researchers at Microsoft classified 570 real review comments, only one in seven was about a defect. Comments suggesting improvements were twice as common. At Google, when engineers were asked about the comments on one specific change, only two of the 44 who answered said the review had found a bug.

Review was never mainly a bug filter. It was a bundle of jobs that happened to travel together while people wrote all the code. Agents pull that bundle apart, and that changes what a team should ask of its reviewers.

When agents write most of the code, machines should take over most of the defect catching, people should spend their reading where machines are weakest, and how deeply they read should depend on what a change can break, set by rules the agent cannot lower. Do that, and the bottleneck stops being reviewer time in general and becomes the few people who can judge the dangerous changes.

Four jobs in one

Code review did four things at once.

It caught defects, occasionally. It kept the design coherent, because someone who knew the whole system could see when a change duplicated a helper or bypassed a layer. It shared knowledge, because the reviewer learned the code and the author learned from the comments. And it put a second name on the change, so a person was accountable for what shipped.

An agent as author changes each job differently. Defect catching shifts most, because automated checks can run on every change and most defects a machine can detect never need to reach a person. Design coherence gets harder, because an agent has no stake in the system's shape and writes changes that are right locally and redundant globally. Knowledge sharing changes shape: the author learns nothing from the comments, and the reviewer may be the only person who ever reads the change. Accountability doesn't change at all, and matters more, because an agent can't be answerable for anything.

What's different about reading an agent's code

The usual warning signs disappear. Reviewers decide where to look from cues: awkward naming, a hedging comment, a messy change that suggests a confused author. Agent output has tidy formatting and confident comments whether or not the logic is right. That is a hypothesis about reviewer behavior, but a related effect has been measured: in a small Stanford study with an early code model, people using the assistant wrote less secure code and were more likely to believe it was secure. Plausible output can raise confidence faster than correctness.

The errors that remain are about meaning. Agents run compilers, linters and tests in their own loop (How a Coding Agent Actually Works), so what reaches a person already compiles and passes its tests. It can still solve the wrong problem: an assumption the request left open, an adjacent API, a rule that holds only for the tested inputs (Tests Are the New Specification). Design problems pass every test too.

There is much more to read. Faros AI, which sells engineering analytics, reports that under heavy AI adoption pull requests, the proposed changes submitted for review, were about half again as large, and median review time was about five times longer. A faster step upstream doesn't shorten the job when a slower step downstream sets the pace (Where the Saved Time Goes).

The author is always available and not fully reliable. An agent can summarize its intent, assumptions and risks in seconds. But the summary comes from the same context that produced the code, so it can leave out a decision or describe something slightly different from what was written. Treat it as a claim to check.

Comments no longer teach. A person told once that the team wraps database calls in a helper tends not to repeat the mistake. An agent starts its next session without the comment. A review comment that doesn't become a test, a lint rule or a line in the agent's instruction file will be written again next week. Turning each real failure into a permanent check is a ratchet, and review comments are one of its best sources.

Review catches fewer defects than people expect and does work that tests cannot. Both facts shape what to keep with people.

Let machines go first

By the time a person opens a change, everything a machine can decide should already be decided, with the evidence attached. Roughly from cheapest to dearest: the build, type checks, linters and formatters; the test suite, with new tests checked for meaning by mutation testing, which deliberately breaks the code to see whether the tests notice; static analysis, secret scanning and dependency checks, including a check that every imported package really exists, since models invent plausible package names; and an AI reviewer in a fresh context that sees the change and the requirements but not the reasoning that produced the code.

AI review is now a standard product, and vendors disagree about what it may decide. Anthropic's Code Review runs several agents over the change, checks its findings against the code to filter false alarms, and never approves or blocks a pull request. GitHub's Copilot code review comments by default, but in a public preview an administrator can let its approval count as the required review, the way a teammate's would. Neither vendor publishes a detection rate, and no independent measurement of how many serious defects AI reviewers miss on real production changes is publicly available.

So machine review belongs in the process as a filter on what people read, with a person's sign-off kept as the accountable step wherever the risk calls for one.

What people should read

Human attention should go where machines check poorly. Five places qualify.

Whether the change solves the right problem. The tests encode the request, and the agent wrote its code against the same request. Someone who holds the original need can see when a faithful implementation misses it. Ask which decisions the request left open, which way the agent went, and what the change leaves out, because an absence never shows up in a diff.

Whether it fits the design. A duplicated helper, a layer bypassed, a new dependency, a public surface that grew. Each passes the tests; judging them requires knowing what the codebase contains and where it is heading.

Security and data handling. Scanners find known patterns. They don't know which input is trusted, what a log line will expose, or who may call a new endpoint.

Irreversible operations. Migrations that drop or rewrite data, deletions, messages to customers. The questions are what happens if this runs twice, stops halfway, or meets production data unlike the test data, and what the way back is (Reversible Work and Irreversible Effects). Review the plan and the rollback before the code.

Whether the tests mean anything. A change that edits a function and its tests together deserves the closest reading of anything an agent produces.

Depth by risk, set by rules

Review depth should follow what a change can break, not how many lines it changes. A one-line edit to a permission check can matter more than a 2,000-line generated client.

Two rules make that workable. First, the risk tier comes from rules about what the change touches, never from the agent's own estimate: a file that assigns owners to paths in the codebase, detectors for migrations, new endpoints, dependency changes and edited tests, and a rule that a change takes the highest tier of anything it touches. The agent may ask for a higher tier and can never lower one.

Second, whoever owns the tiers checks that they hold. A random sample of low-tier changes gets read anyway, and every defect found after release is traced to the tier that let it through (How AI-Native Teams Organize). Without the sample, reviewers drift into automation bias, approving what looks familiar, and nobody learns that the low tier has stopped being low risk.

Scroll sideways to see the whole figure

Sixty pull requests a day

Here is what tiering does to a team, worked through with numbers that are invented to show the mechanism.

A team receives 60 agent-written pull requests a day: 12 trivial ones (documentation, added tests), 30 changes to internal logic with good tests, 10 that touch a public interface or edit tests, 5 security-sensitive ones and 3 migrations. Count only serious defects that survive the automated gates, and assume the riskier tiers carry more of them: about 1.8 a day in total reach reviewers. Assume a careful reviewer finds most of what they'll find early, so 10 minutes catches about 35 percent of serious defects, 30 minutes about 70, and an hour or more close to 90. The team has 26 reviewer-hours a day: five generalists and two specialists in security and data.

The tiered policy spends 3 minutes on the trivial tier, 10 on internal logic, 30 on public interfaces, an hour on security and an hour and a half on migrations.

A few more get through: about 0.64 a day, against 0.55 for reading everything for half an hour. Reading everything for an hour would let only 0.26 through, at 60 hours a day the team doesn't have.

Scroll sideways to see the whole figure

Three things in this example are worth more than the numbers.

Tiering buys most of the protection for less. To match the tiered policy's escape rate with uniform review, you'd need about 25 minutes per change, roughly 25 hours a day. Tiering saves about a fifth of the reading. And returns diminish fast: going from 10 to 30 minutes a change costs about 32 reviewer-hours for each extra defect caught; going from 30 to 60 costs about 106.

The cheap tier carries most of the leakage. Internal-logic changes are individually safe, but there are 30 a day, and they account for about 61 percent of the defects that escape. Extra minutes there return little. Better automated gates and the sample audit return more: an AI pre-reviewer that independently caught 35 percent of serious defects would cut escapes from 0.64 to about 0.42. The word "independently" carries the weight, because a reviewer from the same model family as the author may miss the same things.

The bottleneck moves. Under uniform 30-minute review everyone is overloaded: 30 hours of reading against 26 of capacity, and a queue that grows without limit. Under tiering, the generalists use about half their time, and the two specialists are asked for 9.5 hours against 6 they have, 158 percent of capacity. The scarce resource is no longer reviewer time in general. It's the few people who can read a migration or an authorization change. Ignore that, and specialist changes wait or specialists skim, and the careful tier becomes a rubber stamp.

There are three responses. Give specialists evidence so they need fewer minutes: a dry-run report for migrations, scanner output for authorization changes. That alone brings their load down to about 93 percent, still a long queue. Add a third specialist, which with that evidence brings them to about 62 percent. Or change the architecture so security-sensitive and irreversible code lives in a few marked places, and fewer changes land in the top tiers at all.

The cheap tier carries most of the leakage, and the scarcest reviewers set the pace.

Who signs

An approval says a named person accepts responsibility for the change. An agent can't say that, and a person who clicks approve on something they didn't read has said it falsely.

The workable rules are the old ones. For public interfaces, security and migrations, a named human reviews, and that person is not the one who prompted the agent, which is segregation of duties. For the low tiers, the automated gates and the sample audit carry the weight, and whoever owns the tier rules answers for them. That's also how to read GitHub's option: an AI approval can reasonably count in a low tier the audit covers, and should be switched off for the high ones.

"AI reviewers will catch everything"

The strongest objection runs like this. People have never been good at finding defects by reading. Reading agent code at volume is tedious, and tired reviewers rubber-stamp. A model reads every line with the same attention at a falling price, and a human approving code they don't understand is performing a ritual.

Part of that is right. For style and common local bugs, a machine reviewer is likely to beat a tired human, which is why the design above gives it the first pass. It fails where it claims everything. No vendor publishes a detection rate to support it. A reviewer from the author's model family shares its training and its assumptions about what the code was for, and models tend to rate their own outputs higher. The hardest defects are mismatches with an intent that lives in people's heads. And two of the four jobs, accountability and shared understanding, aren't capability problems. A more accurate reviewer doesn't make a company answerable to its customers.

What to do

Build the gates before you scale the agents. In a codebase with weak tests and slow builds, the low tier isn't low risk. Fix that first.

Set tiers by rules, and let the agent raise them but never lower them. Then audit a random sample of low-tier changes and trace every escaped defect to its tier.

Count your specialists. Before adding agents, estimate how many high-tier changes per day your security and data people can read properly. That number, more than total reviewer hours, sets how fast you can safely go.

Make review cheaper per change. Ask agents for small, stacked changes so each piece gets the right tier: the migration alone, then the logic that uses it, then the interface. Review the plan before the code for anything irreversible. Ask for evidence (test output, command results) instead of claims, and check one stated decision against the diff.

Turn every repeated comment into a check. If you've written it twice, it belongs in a test, a lint rule or the instruction file.

Keep reviewers able to review. Reviewing agent code well takes knowing how code fails, which came from writing it, and heavy delegation can erode that. Rotate who reads the high tiers, and now and then seed known defects into the queue to see whether anyone notices.

Check yourself

Why is the agent's own summary of its change not enough?

It comes from the same context that wrote the code, so it can leave out a decision or describe something slightly different. Treat it as a claim to check.

Why must the risk tier come from rules and not from the agent?

An agent that sets its own tier could lower it. Rules about what the change touches hold, and the agent may only raise a tier.

In the sixty pull requests example, what became scarce?

Specialist time. Tiering made reading cheaper overall, but the two specialists were asked for more hours than they had.