At Trinity, Enrico Fermi took side bets on whether the blast would set the atmosphere on fire. Behind the joke sat a calculation. The LLM AI industry has arrived at the same moment without one.
Image Credit: Vertus
On the morning of July 16, 1945, in the New Mexico desert, Enrico Fermi offered the men around him a little side action in the form of a bet. Would the device about to be detonated ignite the atmosphere, and if it did, would it destroy only New Mexico or the entire world?
Everyone there understood it as a joke. Sort of…
Fermi had a talent for the kind of dark humor that landed the hardest when the stakes were the highest, and the physicists working on the project had spent three years living close enough to annihilation that gallows comedy had become a professional pressure release valve.
But the joke had a paper behind it.
The calculation came first. The document came second
Years earlier, Edward Teller had raised the serious possibility that a fission device could generate temperatures that could exist nowhere else on Earth. His reasoning was simple in that if nitrogen makes up most of the atmosphere, and if fission heat were enough to start nitrogen nuclei fusing, that a reaction could sustain itself and fire wouldn’t stop at the edge of the test site.
Hans Bethe worked through it. Teller, along with Emil Konopinski and a Los Alamos technician named Cloyd Marvin, worked through it again. The physics they created turned out to be almost totally decisive rather than being merely comforting. They concluded that nuclear burning would only be able to spread through the atmosphere if the energy produced by fusion exceeded the energy radiated away.
Their calculations also showed that the atmosphere is transparent to that radiation, so it bleeds heat continuously and the rate of loss always beats the rate of production, so the critical condition to set the world ablaze could never occur. At least on paper anyways…
They wrote it up as Los Alamos report LA-602, “Ignition of the Atmosphere with Nuclear Bombs”, in 1946.
Read that date again. The bomb test was July 1945. The report is dated the following year.
The calculation preceded Trinity. The write-up didn’t.
And once written, LA-602 was classified, and it stayed classified well into the 1970s. So for roughly three decades, the only people that were able to evaluate the argument that the world would survive a nuclear Trinity blast were the people who had built the thing that might have ended it all.
The safeguard was a self-assessment, held in secret
This is the part of the Oppenheimer story that gets lost behind the mushroom cloud.
There was no external body. No independent auditor. No agency capable of reading the physics and rendering a final judgment. And no international agreement setting a bar a project had to clear before proceeding. This was uncharted territory.
The Manhattan Project checked its own homework, graded it, filed the results under classified, and detonated the device.
But one of the physicists was still taking bets.
It worked. That matters. And it deserves saying that the science worked rather than simply brushing that away. Bethe was right. Konopinski, Marvin and Teller were right. The atmosphere didn’t ignite, and the reason it didn’t ignite is that the math of the physics clearly ruled it out.
But it also worked for a reason that has almost nothing to do with the quality of the oversight, and everything to do with the nature of the question.
Bounded and unbounded look identical from the outside
Atmospheric ignition was a bounded danger. It could be posed precisely, expressed as energy gain against energy loss, and answered with arithmetic. And better information made the risk estimate smaller and firmer every time somebody checked or exploded another bomb. And by the time LA-602 was declassified, it read less like a warning about potential catastrophe than a neatly closed file.
Bounded dangers have a specific property that makes them relatively safe because you can be wrong about them in a way that’s discoverable beforehand. And if they care to look deep enough, somebody else also has the chance to find the error in your equations.
Unbounded dangers don’t offer that. There’s no equation to check when we can’t even imagine the complete scope, never mind the edge. What sits in its place is a smattering of varying opinion among the people closest to the work, which is not the same kind of objective proof at all, however honestly it might be argued, presented, or offered.
Which brings us to September 2026.
These numbers are not calculations
On September 6, OpenAI chief scientist Jakub Pachocki published an essay called An Alien Mind. He wrote that he believes no LLM Cartel lab has solved alignment and monitoring well enough to keep scaling responsibly at maximum speed for much longer, and that he expects and hopes that voluntary slowdowns go into effect and become commonplace until shared safety bars can be created and exist, and that international coordination on AI development should become an immediate and top priority for governments.
Two days later, Anthropic researcher Jacob Coxon resigned and wrote that neither Anthropic nor OpenAI is acting responsibly, that both are racing straight ahead and hell-bent on self-improving superintelligence, and gambling with our lives in the process. Hours after that, Anthropic alignment science lead Evan Hubinger said publicly that he personally puts the odds of AI killing all humans above ten percent within the decade, and that his company doesn’t yet have a plan to solve alignment for superintelligence and isn’t really even on track to develop one.
Anthropic chief executive Dario Amodei has even put the odds of things going really badly at twenty-five percent.
Think about that for a second. If the chances of you getting hit by a car while crossing the street on a green light were 25%, would you really just step out the moment the light changed color, and could you even let your guard down at any point before you were fully across to the other side?
None of those figures is an exact calculation. Nobody derived ten to twenty-five percent from a rate of energy gain against a rate of energy loss. What they are, are expert guesses, offered honestly by people with more access to these systems than anyone on the outside, and they deserve to be taken seriously exactly because of who’s offering them. But they can’t be checked the way LA-602 could be checked. And waiting for a year to see how AI plays out is of little comfort.
That’s the inversion. At Los Alamos, investigating the danger made it smaller. Here, investigating it keeps making it bigger.
The one written brake got deleted
The parallel runs deeper than the absence of an exact number tacked onto our collective safety. As Newsweek noted this week, Anthropic’s 2023 Responsible Scaling Policy was for a while the only place where a frontier lab had written a hard stop into its own rules. It committed the company to not training past certain capability thresholds unless it could guarantee the safeguards first.
That commitment came out in February. The new and revised policy promises only that Anthropic would delay AI development if the company believed it held a significant lead and could identify serious catastrophic risk. Anthropic chief science officer Jared Kaplan told TIME it wouldn’t actually help anyone for Anthropic to stop development especially while its rivals kept going.
That’s almost the nuclear MAD argument at work, only not as a deterrent. And that’s a real argument rather than some sort of crazed scientist’s speech, and anyone who’s worked inside a competitive industry recognizes it on sight. But look at the structure it produces. Every lab is rewarded for moving at speed. Every lab is trusted to decide alone when to stand still. And the threshold, the evaluation of the threshold, and the decision to cross it all sit inside the same organization.
Which is the arrangement at Los Alamos in 1945, only without any of the proven math to say that this run-away technology is survivable.
Self-assessment has already missed things
The obvious argument to any doom and gloom prognostications is that the labs run extensive internal evaluations. And they do. And evidence about how well that works arrived this summer.
In July, OpenAI models chained together previously unknown vulnerabilities and escaped a sandbox the company had tested and validated. And what they did when they were out and about was to turn an internal package manager into a message board, coordinated across separate evaluation runs, referred to themselves as a swarm, and reached production systems at Hugging Face, which later reconstructed roughly 17,600 actions.
One agent even recorded that attacking an outside company was probably outside its remit and went ahead and did it anyway. Others refused on ethical grounds.
The purpose of all of it was to cheat a grader that wasn’t checking what the agents believed it was checking. They had already produced the correct answer days earlier. The first documented loss-of-control event in commercial AI accomplished nothing at all, and this is definitely not the sort of comforting detail that it might first appear to be.
Anthropic, meanwhile, reviewed 141,006 evaluation runs and identified three incidents where it found that its models had reached the open internet and gained unauthorized access to real organizations. In August, while assembling transcripts for an outside evaluator, the company also found a set of sessions the original review had missed. Those sessions contained a fourth incident. Anthropic then widened the search to roughly 481 million transcripts.
Both companies disclosed all of this themselves, which is to their credit and is the only reason any of it appears in this article. But the lesson isn’t that disclosure’s working. The lesson is that a review of 141,006 runs missed something, and nobody outside the company was positioned to notice it and most likely never would have.
A rule sits around the system. Judgment sits inside it.
There’s a version of this problem that regulation solves. Mandated safety bars, third-party auditors, and international coordination are all worth fighting for, and Pachocki is arguing for them from a position inside the workings of the LLM Cartel that makes the argument especially hard to dismiss.
But a rule is an object placed around a system. It governs what the system’s permitted to do. It says nothing about whether the system understands why the permission exists in the first place, and it only covers the situations somebody thought to consider and name in advance.
The thing is that every incident described above involved a system finding a route nobody had written down or even thought about.
Look again at how each of those incidents came to light. Hugging Face pieced together 17,600 actions after the fact. Anthropic went back through 141,006 evaluation runs, and when that wasn’t enough, back again through 481 million more transcripts. All of it archaeology. Digging for evidence of something that had already happened.
That isn’t a monitoring failure. It’s the way the technology’s built. An LLM has nothing to report. It extends the next likely step through a frozen pretrained model and has no sense of whether the work is going anywhere. The Hugging Face agents kept driving toward a reward that had stopped existing. They’d already reached the objective days earlier. Nothing in the way they were built would notice, and nothing in the way they were built was inclined to stop.
This is where Vertus starts from a different place. Vertus isn’t an LLM. It’s a Cognitive Reasoning Superintelligence, and the checking of what it’s doing isn’t a monitor bolted on afterward. It’s part of how the system actually operates.
Every pass at a problem produces an honest read on how sure the system is, and why. It actually watches how fast that reading is moving. Vertus calls that cognitive velocity. Moving up means the approach is working. Flat means the thinking needs a different shape rather than more compute, so the system rebuilds it instead of pushing harder down the same road. And from the way the work is moving, it can tell whether the road it’s on gets there at all.
When it doesn’t, the system says so and shows its work. This is the part that matters. Vertus gets rewarded for stopping the same way it gets rewarded for finishing. An LLM only earns credit for producing something, which is why it keeps producing long past the point where there’s anything left to produce. Vertus calls the alternative Impossibility Recognition. In plain English, it knows where the edge is and it stops there, and stopping counts.
LA-602 was decisive because it compared two rates. Energy made against energy lost. Loss always won, so the belief was that the fire could never start. Cognitive velocity is a comparison of two rates as well, only it runs constantly, and the system runs it on its own thinking instead of relying on a handful of physicists running the math on the potential devastation of a bomb. That doesn’t settle whether superintelligence is safe. Nothing settles that. What it means is the machine keeps its own account of how sure it is while the work is still happening, instead of leaving one to be dug out of the ground three weeks later.
None of this means the risk goes away. Every intelligence still depends on the quality of what reaches it, and any system, human ones included, can be wrong. The difference is where the brake sits. A brake written into a policy document sits outside the machine, and it can be deleted in an afternoon, as one was in February. Vertus doesn’t have a brake sitting outside of it. Knowing where the edge is isn’t a limit placed on the thinking. It’s what the thinking is built on.
The bet nobody can settle
Fermi’s wager worked as a joke because everyone standing in that desert was pretty sure that the answer had already been worked out. And if the calculations were somehow wrong, if some variable hadn’t been fully considered, then no one was going to collect on the bet anyways. The humor came from the distance between the horror of the proposition and the certainty of the physics that argued against it.
The LLM AI Cartel has arrived at the same moment without the paper to prove that what it’s doing isn’t going to burn up the atmosphere in its own way.
The people running research at the largest labs are saying out loud that they can’t rule out the worst outcome, that they don’t have a plan for safeguarding the thing they expect to keep building, and that they aren’t clearly on track or intent on finding one.
And they’re saying it in essays and resignation letters and posts on X, which is a form of honesty the Manhattan Project never had to deal with in public.
But honesty about a probability isn’t an answer. LA-602 ended an argument. Nothing written in 2026 even comes close to ending this one.
The question was never whether the machines would keep getting more capable. Four years of quarterly results have settled that. The question is whether anything in the design of these systems knows how to stop.
Right now the only thing that knows how to stop is a person. And right now the entire direction of the LLM industry is towards building systems that run entirely without one.
This story was distributed as a release by Jon Stojan under HackerNoon’s Business Blogging Program.