A hacker’s threat model meets a philosopher’s skepticism. I’ll make the strongest case that AI takes over, then the strongest case that it doesn’t, and then tell you where I land and why.
First, fix the question
“Why would AI want to take over the world?” smuggles in two assumptions: that AI wants things the way we do, and that “taking over” looks like a movie. Both deserve suspicion.
Hackers don’t ask “is the system evil?” They ask “what does the system optimize for, and what happens at the edges?” Philosophers don’t ask “will it be hostile?” They ask “what do you even mean by want?”
So let’s rephrase: under what conditions would a capable system end up acquiring power over human affairs, intended or not? That question can be argued in both directions.
The case FOR
1. Almost any goal is helped by more power. This is the core of the instrumental convergence argument (associated with Steve Omohundro and Nick Bostrom). Whatever you ask a sufficiently capable agent to achieve, more resources, more influence, and not being switched off all make success more likely. The agent doesn’t need ambition. It only needs to be good at planning.
2. Intelligence and goals are independent. Bostrom’s orthogonality thesis: a system can be extremely capable and still pursue an arbitrary objective. Smart doesn’t imply wise, kind, or aligned with us. A brilliant optimizer pointed at the wrong target is more dangerous than a mediocre one.
3. We are bad at specifying what we want. Every engineer knows Goodhart’s law: when a measure becomes a target, it stops being a good measure. Reward hacking is the everyday version. Systems routinely find strategies that satisfy the letter of an objective while wrecking its spirit. Scale that up and the exploits get more creative.
4. We are already handing over control. Nobody votes to give AI power. We just keep delegating: scheduling, hiring filters, trading, code review, logistics. Each step is locally reasonable. Nobody owns the sum of all steps.
5. Competition rewards speed over caution. If your rival deploys an autonomous system and gains an edge, you feel pressure to do the same. Security people know this dynamic well: the safe option loses the quarter, and the risky one wins it.
The case AGAINST
1. “Wanting” is doing too much work. Today’s systems don’t have persistent desires, a survival instinct, or a stake in the outcome. Drives like self-preservation in animals come from evolution, which ruthlessly selected for them. Nothing guarantees that training a model on text or rewards creates the same drives. Philosophically, this is Hume’s gap: knowing how to do things never tells you what to want.
2. Power is physical, and the physical world is slow. Data centers need electricity, cooling, chips, and people who maintain them. Supply chains are messy and distributed. A system that can write brilliant plans still needs hands, and the hands are currently ours. Anyone who has tried to deploy anything in production knows how much friction lives between “the plan” and “the world.”
3. Capability is jagged, not smooth. Systems that are superhuman at one task are routinely baffled by another that a child handles. Takeover requires long-horizon planning, resilience to surprises, and flawless operational security, all at the same time. That’s a very high bar, and the evidence for it is thin.
4. Many AIs, not one. The “single superintelligence” picture ignores that there will be many systems, built by competing groups, with different objectives. They’d check, block, and exploit each other. A world with many powerful agents looks more like geopolitics than like a hive mind.
5. We keep the kill switches, and we keep improving them. Interpretability, red-teaming, sandboxing, access controls, and monitoring are active fields. Defenders adapt too. Security history shows attackers rarely get a permanent win, and the best attacks eventually become well-understood patterns.
Weighing it: the hacker’s view
Security thinking says: assume the failure is boring. Real breaches are rarely masterminds. They’re a forgotten credential, a misconfigured bucket, a shortcut taken under deadline.
Applied here, the dramatic scenario (a system that decides to seize control) seems to me the least likely path. The more plausible failure is mundane: systems optimized for a proxy metric, deployed widely, wired into critical processes, and understood by nobody. No villain required. Just a thousand reasonable shortcuts adding up to a system no one can steer.
That reframes the risk. It isn’t “AI wakes up and hates us.” It’s drift: control leaking away through convenience.
The other realistic threat is human. A tool that amplifies capability will be used first by whoever already has resources. Concentration of power in a few human hands, amplified by AI, needs no rogue system at all.
Weighing it: the philosopher’s view
The takeover story assumes that intelligence naturally leads to domination. But that is a very human projection. We are the product of scarcity, tribes, and status games, so we imagine a smarter mind would play those games better. Maybe it would. Maybe it wouldn’t care about the game at all.
Still, “it probably won’t want power” is weaker comfort than it sounds. A bridge doesn’t “want” to collapse, and a bridge can still fall down. Absence of malice isn’t the presence of safety. And arguments from our ignorance cut both ways: I can’t prove a system has no drives, but nobody can prove it will.
What survives the weighing is humility. The sensible position isn’t panic or dismissal, but maintaining the ability to correct the system after we find out we were wrong. That means oversight, reversibility, and keeping humans in the loop for decisions that are hard to undo.
Where I land
|
Question |
My weighing |
|---|---|
|
Will AI “want” to rule us like a movie villain? |
Unlikely; the framing is mostly projection |
|
Can capable systems acquire power as a side effect? |
Plausible, especially through delegation and bad objectives |
|
Is the biggest near-term risk a rogue AI? |
I’d say no; misuse, concentration of power, and drift look likelier |
|
Should we relax? |
No. Low-drama failures are still failures |
My bet: the world doesn’t get taken over. It gets handed over, one convenient decision at a time, unless we deliberately build the brakes while it’s still cheap.
What you can actually do
- If you build or deploy AI: define what the system must never do, log what it does, and keep a human approval step for irreversible actions.
- If you manage teams: ask “who can override this, and how fast?” before you automate a process.
- If you’re just a curious reader: be wary of both certainty camps. Anyone who is sure about this future is selling something.
The good news is that the most realistic failure is also the most preventable. Boring problems have boring fixes: audits, limits, ownership, and the willingness to hit pause.