The Safe Way to Ship Production Code Written by AI Agents

by

There’s a specific kind of pull request happening right now that would have been unthinkable three years ago: a feature branch opens, tests run, a bot writes the implementation, a human clicks “approve,” and the change ships — and the human never typed a line of the diff. Not reviewed and cleaned up. Reviewed and shipped.

That’s not a demo. GitHub says developers used Copilot coding agent — the version that goes off, plans a multi-step task, and comes back with a finished PR — to create more than a million pull requests between May and September 2025, according to GitHub’s Octoverse 2025 report. Developers merged 518.7 million pull requests on the platform in 2025, up 29% year over year. Something structural changed in how software gets written in 2025, and most engineering leaders are still writing policy for the old thing.

This piece is a playbook, not a hype piece. Agentic coding tools are genuinely useful and genuinely risky in ways that are different from the autocomplete era, and the honest data on productivity gains is a lot messier than the vendor decks suggest. Let’s get into it.

Autocomplete, Copilot, Agent: these are not the same category

The industry conflates three very different tiers of tooling, and the conflation is where a lot of bad governance decisions come from.

Autocomplete finishes the line you’re already typing. It has no opinion about your task, only your syntax.

Assistant-style tools — GitHub Copilot Chat, inline chat in an IDE — answer a question or generate a snippet on request. You drive every step; the model never touches a file unless you paste its output in yourself.

Agentic tools are different in kind, not degree. Give something like Claude Code or Aider a goal — “fix this failing test,” “implement this ticket” — and it plans a multi-step approach, executes tools (reads files, runs the test suite, edits code, checks the result), and iterates without a human approving each intermediate step. The human re-enters the loop at the PR, not at every keystroke.

That shift — from “suggest text” to “operate the toolchain” — is the entire reason the risk profile changed, and it’s the reason “just review the diff carefully” isn’t a sufficient governance strategy anymore. You’re not reviewing a suggestion. You’re reviewing the output of an autonomous multi-step process you didn’t watch happen.

The productivity numbers are genuinely contested — and that’s worth sitting with

Here’s where I want to push back on the narrative most of these articles run with, because the honest picture is more interesting than “AI makes you faster.”

In July 2025, METR — an independent AI evaluation nonprofit — published a randomized controlled trial that found the opposite of what everyone expected. Sixteen experienced open-source developers, working on mature codebases they knew intimately (about five years of prior experience on average), were randomly assigned to complete real GitHub issues with or without AI tools (primarily Cursor Pro with Claude 3.5/3.7 Sonnet). Before the study, developers predicted AI would cut completion time by 24%. Afterward, they still believed it had made them roughly 20% faster. The actual measurement: they were 19% slower with AI allowed. The paper is titled “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity,” and METR published the full write-up along with the arXiv preprint.

That finding traveled everywhere, and for good reason — a 39-point gap between perceived and measured performance is not a rounding error. But two things are also true and matter for anyone using this number to make a policy decision:

First, the confidence interval on that 19% figure was wide — roughly +2% to +39% — and the study population was narrow: sixteen people, on codebases where the developer already knew the code better than any AI tool could. METR was explicit that the slowdown shouldn’t be read as a universal verdict; it’s a measurement from a specific, unusually hard setting for AI to add value.

Second, METR tried to run the experiment again with a larger, newer cohort using current tools, starting in August 2025. But by February 2026, they reported a problem that’s arguably more interesting than the original finding: 30–50% of invited developers were declining to participate because they didn’t want to work without AI access. That’s a selection effect quietly eating the study from the inside — the developers most willing to go without AI for a randomized trial are, almost by definition, the ones who feel they get the least out of it, which biases any “no-AI” baseline. METR’s own conclusion was that the resulting data gives only weak evidence about the actual size of any speedup, and they’re redesigning the experiment rather than standing behind a new headline number.

So the follow-up didn’t simply reverse the original finding — it exposed a second problem underneath it. Once AI tools become normal enough that skilled developers actively resist working without them, measuring the true counterfactual gets harder, not easier. That’s a more interesting methodological problem than “did the number go up or down,” and it’s a sign of how fast the ground is moving under any productivity claim you read right now, including the ones in this article.

The honest summary: the tools got better, fast, between February 2025 and today, and single-task completion speed is probably the wrong metric anyway. A separate analysis from workflow-analytics firm Faros AI found that AI-assisted teams showed a 21% higher task completion rate and 98% more merged PRs, driven less by individual tasks completing faster and more by developers parallelizing — kicking off one piece of work with an agent while reviewing another. Organizations don’t actually optimize for how fast one developer finishes one ticket; they optimize for shipped, working software across a team. Those are different curves, and the METR study measured the first one on purpose.

Take the productivity claims from any tool vendor with real skepticism until you see the methodology. And when someone in an all-hands says “we’re 40% faster now,” it’s worth asking: faster at what, measured how, compared to what baseline.

What’s actually happening under the hood of an agent

An agentic coding tool isn’t the model plus a chat window — it’s the model wrapped in what practitioners call an “agent harness”: scaffolding that lets a model act on a codebase instead of just describing what to do.

Three components matter:

  • Context retrieval — code search and indexing so the agent can find the three relevant files in a five-hundred-thousand-line repo instead of guessing, or worse, hallucinating a function signature that doesn’t exist.
  • Tool execution — the actual permission to run shell commands, edit files, and execute the test suite, not just describe what those commands should be.
  • A verification loop — did the tests pass? If not, revise and retry, potentially several times, before ever surfacing a PR to a human.

The loop, in practice, looks like: read the task → retrieve relevant context → propose a change → apply it → run tests → pass, open a PR, or fail, revise and try again.

Developer
   │
   ▼
Task / Goal
   │
   ▼
 Agent  ──► generates a hypothesis for the change
   │
   ├──► Repository Search
   ├──► File Editor
   ├──► Shell
   └──► Test Runner
           │
           ▼
     Verification  ──► the system observes whether the hypothesis held
        /        
     Fail        Pass
      │            │
      ▼            ▼
    Revise        Open PR

It’s worth being precise about what that loop is actually doing, because it’s tempting to describe it as “the agent tests its own work, so hallucination gets caught.” That’s not quite right. The model generates a hypothesis, the tools test that specific hypothesis, and the system observes the result — pass or fail — and revises accordingly. What it does not do is verify that the hypothesis was the right one to test in the first place. A test suite only checks what it was written to check; an agent can satisfy every assertion in a test file while doing something structurally wrong that nobody wrote a test for. The loop is real and it does catch a meaningful class of errors. It is not proof of correctness, and treating it as one is exactly the mistake that makes the next section necessary.

The guardrails that actually matter

This is the part that determines whether adopting agentic tooling makes your engineering org safer and faster, or produces a slow-motion incident. Five things, in order of how often teams skip them.

1. Scoped permissions, enforced at the credential level. The agent gets write access to the branch it’s working on. Not production infrastructure, not secrets, not other repos. This has to be enforced by IAM and token scoping, not by a line in the system prompt asking it nicely not to touch prod. Prompt instructions are not a security boundary.

2. Automated tests as a non-negotiable merge gate. An agent’s change doesn’t merge without the existing suite passing, plus new tests for new behavior. This is the primary defense against silent regressions and confidently-wrong logic, and it needs to be a CI status check, not a request.

But be honest with your team about what “tests pass” actually proves — which, on current evidence, is less than most people assume. METR published an analysis in March 2026 where maintainers from three real open-source projects — scikit-learn, Sphinx, and pytest — reviewed 296 AI-generated pull requests that had already passed SWE-bench Verified, a standard benchmark for coding agents. Roughly half of those test-passing PRs would not have been merged into the real codebase. The maintainers’ reasons weren’t nitpicks: core functionality failures, code that broke other parts of the system, and straightforward code-quality problems the test suite had no way to catch. The lesson isn’t that automated tests are worthless — it’s that a test suite verifies only what someone thought to write a test for. A maintainer evaluates architecture, scope, and whether a change actually belongs in the codebase, and none of that shows up in a green checkmark. Passing tests is necessary. It has never been sufficient, and agentic tooling makes that gap between “tests pass” and “a human would approve this” wider and more consequential than it was when a human wrote every line.

3. The same static analysis and SCA scanning you’d run on human code. Agent-written code introduces the same vulnerability classes humans do, and a newer one specific to this generation of tools: dependency hallucination, where an agent references a package name that doesn’t exist — which is now a documented supply-chain attack vector, since someone can register that exact hallucinated package name with malicious code inside it.

4. Human review before merge, always. Not because you distrust this particular model — because you’d require review for any new contributor, human or not, and an agent is, functionally, a very fast, very inconsistent new contributor.

5. Policy-as-code enforcement in CI, applied identically. License compliance, banned patterns, dependency allowlists — the same automated checks that gate a human’s PR should gate an agent’s, with zero special-casing.

None of these auto-apply changes without a human in the loop. That’s the whole point: propose, gate, review, merge — not propose-and-ship. In practice, the pipeline looks like this, and every stage below is a hard gate, not a courtesy check:

Agent-authored PR
      │
      ▼
  Unit / Integration Tests
      │
      ▼
  SAST (static analysis)
      │
      ▼
  SCA (dependency / license scan)
      │
      ▼
  Secret Scan
      │
      ▼
  Policy-as-Code Check
      │
      ▼
  Human Review  ◄── the only stage evaluating architecture, scope, and "should this exist"
      │
      ▼
     Merge

And the permission scoping from guardrail #1 is worth making concrete rather than abstract. Below is an illustrative configuration pattern — not the actual syntax of any specific product like Claude Code or Copilot, which each have their own config formats — meant to show the shape a scoped agent setup should take: a job description with hard limits written into the credentials, not the prompt:

agent:
  goal: "Fix failing test in checkout-service#4821"
  repo_access:
    - repo: checkout-service
      branch: agent/fix-4821      # never main, never a shared branch
      write: true
  tools:
    - repo_search
    - file_editor
    - shell            # scoped to the repo's own sandboxed container
    - test_runner
  forbidden:
    - production_credentials
    - infra_provisioning
    - secrets_access
  verification:
    require_pass: [unit_tests, integration_tests]
    max_revise_attempts: 3
  on_success: open_pull_request   # never: auto_merge, deploy

That last line is the entire argument in one word: open_pull_request, not auto_merge. Everything upstream of that line is about making the agent capable. Everything at and after it is about keeping a human as the last checkpoint before code that can’t be un-shipped goes out.

What actually changes for the humans

The lazy version of this section says “developers become prompt engineers.” That’s not what I see happening. What’s actually shifting is where judgment gets applied: more time on system design and writing acceptance criteria precisely — because an agent will execute literally what it’s told, gaps and all — and more time reviewing and verifying output. Less time on boilerplate implementation.

There’s also a new job showing up on platform and DevEx teams that didn’t exist two years ago: someone owns the agent’s guardrail configuration — what it can touch, what tests gate it, what happens on failure — the same way someone owns your CI/CD pipeline config today. If nobody owns that, it drifts, quietly, until an agent has more access than anyone intended.

When not to use one

Agents are strongest on well-scoped, verifiable work: a bug fix with a reproducing test case, a feature that follows an established pattern in the codebase. They’re weakest on ambiguous, judgment-heavy work — novel architecture decisions, anything where “correct” can’t be checked automatically.

A simple heuristic that’s held up for me: if you can’t write a test that would catch the agent getting it wrong, it isn’t ready to run unsupervised on that task. Keep a human in the loop for that class of work, full stop.

A three-step adoption path that doesn’t require faith

  1. Start narrow. Test-writing, dependency bumps, well-defined bug fixes with a reproducing case. Mandatory human review on everything.
  2. Instrument before you expand. Track merge rate, defect rate, and review time on agent-authored PRs versus human-authored ones for at least a month. Not vibes — numbers, the same way METR insisted on measuring instead of asking developers how fast they felt.
  3. Expand only into categories with strong automated verification. Good test coverage is the actual prerequisite for autonomous execution, not how impressive the demo looked in a meeting.

The closing argument

The METR story is the whole lesson in miniature: the developers in that study weren’t lying when they said they felt faster. They were wrong, and confidently wrong, in a way that only a controlled measurement caught. That’s precisely the failure mode agentic coding introduces at the org level if you let vibes substitute for verification — confident, plausible, and quietly off.

Agentic AI doesn’t remove the need for engineering judgment. It moves that judgment earlier — into how precisely you specify a task and how tightly you design the guardrails around it — instead of eliminating it. The teams that get real value from this are the ones treating it like they’d treat any new, very fast, occasionally overconfident contributor: scoped access, mandatory tests, mandatory review, and numbers instead of impressions before scope expands.

Sources

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.