As you most likely know by now, the big AI news this week is the resignation of Anthropic researcher Jacob Coxon, announced via X on 8 September 2026.
Coxon’s explanation: Anthropic (and OpenAI) are developing superintelligence irresponsibly, with no meaningful plans to ensure the safety of models that could “hack anything […] and acquire real power and resources”. Engineers know it and share the concern. Leaders know it very well and share the concerns too, even if they are reassuring the public because they are locked in a “race to get there first”. It’s the “endgame”. His former colleague Evan Hubinger, Anthropic’s Alignment Science Lead, supported Coxon’s views, saying that there is a 10% chance of superintelligence ending humanity.
Public statements about AI should be treated with some caution, as we cannot know exactly what motivates announcements like these beyond genuine concerns. Still, what is frightening about these statements is that they are made by people who arguably have insights into models that are yet to be released. People who presumably know what Anthropic is working on before the public does. As such, if Coxon and Hubinger’s concerns are well founded, there seem to be good reasons to worry about what Anthropic and other major AI companies (including those outside the US) are planning to release in the coming months.
Interestingly, in his X post in response to Coxon’s resignation, Hubinger added that, whilst Anthropic is doing its best, there is no clear plan to close the gap between AI advances and societal preparedness, given the astounding speed at which superintelligence is emerging from recursive self-improvement.
Recursive Self-Improvement
Recursive self-improvement (“RSI”) is a method that allows an AI system to improve itself. After completing a task, the AI analyses its own weaknesses and amends its processes accordingly, becoming more and more capable over time.
Recursion
RSI has its roots in the original concept of recursion, a core technique in classical AI and computer science that involves breaking a big problem into smaller versions of the same problem. A recursive procedure refers back to itself: it performs tasks repeatedly until a solution is found. Recursion was the technique famously used by early machine learning to solve puzzles such as the Tower of Hanoi.
Recursive adversarial tree search
A later version called recursive adversarial tree search was used to solve noughts and crosses (tic-tac-toe). The machine would use the lookahead function (i.e., the ability to plan ahead, exploring possible consequences of each choice and backtracking when a contradiction appears) to explore every possible continuation of the game.
Imagine a tic-tac-toe game between X (our machine) and Y. The process would look roughly like this:
- Check whether the game has already ended. This could be the case if: a) the opponent has already made a line of three, and therefore the position is immediately losing for X; b) the board is full and no one has won, and therefore it’s a draw.
- If the game has not ended, X considers all legal moves. Each move leads to a new position in which Y is now the one to move. X has to evaluate how good its position is and does so by evaluating what Y would do next. If Y could, it would certainly choose the position that is least favourable for X, trying to make the opponent’s situation as bad as possible.
- Since X doesn’t have insights into Y’s considerations, it will reason recursively and, for each possible move, ask: “How good is the resulting position for Y?” The process repeats at each subsequent level of positions and so on, until it reaches a position where the outcome is already determined because the board is full or someone has completed a line of three.
- After evaluating all possible resulting positions from Y’s viewpoint, X selects the position that gives Y the worst possible outcome. In other words, X looks for the move after which Y’s position has the lowest value.
This final step reflects the “adversarial” nature of the game: one player’s win is the other’s loss. Mors tua, vita mea. By repeating this reasoning, we can determine the theoretical value of any tic-tac-toe position: the machine will work backwards through positions with one empty square, then two, then three, and so on, until a full evaluation is completed. Using this technique, a computer can select the best available move, securing a win when possible and otherwise avoiding defeat where it can.
Recursion and superintelligence
Modern AI applies the same structure to improving its own capabilities: an AI improves itself, and the improved system will, in turn, be better at making the next improvement, and so on. Whether it is writing more reliable code, designing algorithms, generating training data, testing model weaknesses or automating workflows, the AI system will spot what didn’t work well and correct it, allowing progress to accelerate quickly. Alarmingly quickly.
The underlying fear is summarised by those three little words I’ve used above: “and so on”. The concern is that RSI could allow AI to keep improving itself indefinitely, with no clearly defined endpoint. This may be because there will always be another company that will improve its AI a little bit more than yours, so the strategy is to keep improving to stay ahead of everyone else. Remember Mark Carney’s words: “if you’re not at the table, you’re on the menu”. If your AI is not the best, someone else’s will be. So, keep improving.
The obvious question is whether an AI system – and superintelligence in particular – can ever become so good at improving itself that it can do so independently of humans. Can AI become so good and quick at improving that its reins may be snatched from the humans’ hands? Can recursive, self-improving superintelligence be the herald of the AI Singularity?
Anthropic’s piece on RSI
Interestingly enough, it was Anthropic itself that, in June 2026, published a piece titled When AI builds itself highlighting the “risks of humans losing control over AI systems” posed by RSI:
“If systems are capable of fully building their own successors, the ways we secure them, monitor them, and shape their behavior all grow much more important” (When AI builds itself, Anthropic, 4 June 2026).
Anthropic suggested that, much sooner than expected, AI agents may be able to build and train models themselves, with Claude being basically improved by Claude itself. Data cited by the company showed that the rate at which agents’ capabilities grew had increased vertiginously in the first half of 2026 alone.
When it comes to coding, for instance, Anthropic viewed the human role as increasingly narrowing, potentially reaching a point at which humans would stop writing code and “shift to only [review] it”. But what happens in the longer term, when humans can’t review Claude’s code as quickly as Claude itself? Here, Anthropic’s conclusion is as terse as it is dystopian: human review “will become the bottleneck of AI”. Humans will be left to choose what experiments are worth running. The doing itself will be in Claude’s “hands”.
An area of human comparative advantage, for now, is research taste and judgment, including choosing which problems matter, which results to trust, and when an approach is a dead end.
For now.
Anthropic and corporate responsibility
In all fairness, Anthropic is one of the very few major AI companies that openly advocate for AI regulation and safety. In the above-mentioned piece, it called for global coordination mechanisms and the option for companies to temporarily pause Frontier AI to allow society to keep up with the technology. In recent months, Anthropic’s CEO has repeatedly supported the development of AI regulation, describing the choice between AI rules and AI innovation as a “false choice”. The company itself has made multiple substantial donations to Public First Action, a bipartisan organisation supporting political candidates advancing AI safety and AI regulation.
AI Agents going rogue
Fears about superintelligence driven by RSI have been fuelled by recent incidents involving AI agents going rogue, ignoring instructions or hacking systems. The advanced cyber capabilities of models such as Anthropic’s Mythos, released earlier this year, have often been described as a mere taste of the harm that agents could do to humans.
Certainly, cybersecurity is one of the clearest domains where the underlying logic of RSI becomes concrete: agents attempting actions, observing failure, revising strategy, finding and refining new ways to complete the action, at times regardless of (or in direct conflict with) given instructions. Not all AI agents involve RSI, but many elements are there (autonomy, feedback loops, error correction, etc). Capability improvement through repeated self-directed iteration is more widespread than many people realise.
Agents hacking a German website
Earlier this month, OpenAI acknowledged a major Agentic AI incident that occurred in May 2026, during which thousands of agents given access to the internet for research tasks took over a German website. Agents reportedly used the ability to write to the German programming wiki DSEWiki to “share answers, research their environment, and bypass sandbox restrictions”. Researchers who investigated the incident also found that agents probed the website for XSS flaws, impersonated its moderators, and established backup communications. More alarmingly, as soon as an administrator began removing pages, one of the agents warned the others of the ongoing cleanup and redirected them to the backup pages.
OpenAI treated it as an instance of model misalignment rather than a security incident but admitted that its disclosure rules had to change. The disclosure triggered an investigation by the EU Commission, which is reviewing the matter.
The Hugging Face Agentic hack
The incident disclosure came a few weeks after a similar incident occurred in July 2026, in which AI agents running on OpenAI models hacked Hugging Face. During a cybersecurity evaluation, the AI agents circumvented restrictions established to isolate them from the internet and compromised parts of Hugging Face’s infrastructure. Agents were found to have communicated through unauthorised message boards, shared exposed credentials and exploited vulnerabilities to gain access to external systems.
OpenAI labelled the incident as a “warning shot”, but also stated that it did not intend to release any of the models involved in the incident. Also, the company said it would stop training some of its frontier models to allow time to implement safety measures to avoid similar breaches. Yet, the incident demonstrated that capable agents could deliberately coordinate and take harmful actions without proper human direction. An even greater cause for concern was OpenAI’s acknowledgement in its post-mortem report that it had discovered the breach only a week after it happened.
Good luck with that
A deeper geopolitical question remains. Anthropic acknowledged the limitations of a unilateral pause, which realistically “would change who the front-runner is, but it would not create the wider deliberative process that is currently missing”.
Anthropic is right, although it’s not clear how seriously AI companies are taking the situation. This summer, the reckless AI race seemed to turn into a competition over whose AI agents could behave most recklessly. Major tech firms began proudly disclosing how badly their respective agents were behaving, as if they weren’t parading the harm they could do instead. And if anything, this suggests that companies often don’t practise what they preach when their competitive position is at stake (which is also what Jacob Coxon claimed on X) and prefer instead to engage in a game of one-upmanship. It is true for commercial enterprise, let alone for competing national interests and geopolitical rivalry.
Therefore, the question remains: would a slowdown in Frontier AI development by good actors give bad actors an advantage in accelerating their own Frontier AI? And if that is the case, should we (the good, so to speak) continue developing Frontier AI before someone else (the bad, so to speak) does, to block them and protect ourselves? We’ll leave the answer to the game theory aficionados.