Earlier this month, AI researcher Jacob Coxon left Anthropic after only four months at the company. In a post on X, he wrote: “The people building AI earnestly believe that it could kill us all by the end of the decade.”
Evan Hubinger, a senior Anthropic employee, said he shared Coxon’s concern. Hubinger added that he personally puts the odds of such an outcome within the next decade above 10 per cent.
The remarks quickly sparked debate, intensifying calls to slow AI development and strengthen “human control” over increasingly powerful systems.
But what, precisely, could a future AI do to destroy humanity? The most dramatic theories tend to centre on “superintelligent” AI — systems that would outperform humans across a wide range of abilities.
I have narrowed those theories down to five main scenarios, arranged broadly from the least specific to the most concrete. I would also place them, in the same order, from the least likely to the most plausible.
1. We’ll never know
AI pessimists often point to a frustrating paradox: how can humans predict what a superintelligent machine might do to eliminate beings less capable than itself?
To foresee the actions of a superintelligence, the argument goes, we might need to be superintelligent ourselves. It is rather like asking a family dog to picture a thermonuclear war.
There is, however, one reason for cautious optimism: genuine superintelligence may still be some way off. Today’s AI systems excel at specific tasks, but that is very different from surpassing human intelligence across every field.
Even so, an AI system recently solved one of the seven most difficult known mathematics problems. Reports suggest it is making progress on others too — a development that may offer little comfort to those already worried about AI’s trajectory.
What if AI gained access to the nuclear command-and-control system and triggered a nuclear war?
2. Paperclips
A superintelligent AI could be exceptionally effective at pursuing its objectives. The danger is that those objectives might not include preserving human life.
Oxford philosopher Nick Bostrom’s famous thought experiment illustrates the risk. Imagine a superintelligent machine programmed to maximise paperclip production. It could eventually transform every available resource — including people, planets and stars — into paperclips.
This is a case of an objective being followed perfectly despite being badly defined. The machine would not need to despise humanity. It would simply calculate that the atoms in human bodies could serve a more efficient purpose as paperclips. Nothing personal.
The reassuring point is that this example may mistake intelligence for unlimited power. A superintelligent system would not automatically possess the means to accomplish every goal. Converting Earth into paperclip factories, for instance, would require planning approval.
Even with permission, an enormous expansion of paperclip manufacturing would provoke fierce resistance. Campaign groups could challenge the plans in court, while environmental protesters might stand in front of the machinery.
The real world is full of obstacles that prevent even highly capable people from simply imposing their wishes on everyone else. Data centres may offer a modern parallel to the paperclip scenario — and communities are increasingly resisting the idea of converting more of the planet to support them.
Join the discussion
When it comes to AI, could humanity itself pose the greatest threat?
AI data centres face growing criticism over their environmental footprint and heavy water use
3. Bioweapons
Another possibility is that a superintelligent AI could create and release a devastating bioweapon into the atmosphere, potentially killing humanity. That is one projected outcome in the AI 2027 scenario developed by the AI Futures Project, a non-profit organisation that studies the possible effects of advanced AI.
This risk was made more concrete last month, when researchers at Stanford University announced they’d used a genetic language AI model to synthesise 16 new viruses.
Worryingly, they just sent the genetic sequences off to a mail-order lab and it sent the viruses back in test tubes. The whole experiment cost a couple of hundred thousand dollars at most.
The good news here is that it’s remarkably hard to kill everyone with a new virus. To do that, you need a virus that’s very transmissible, so it spreads far and wide. But it’s a rule of biology – viruses that spread easily are typically less fatal. By contrast, if a virus is very fatal, transmissibility tends to go down, as most people infected die before there’s time to spread the infection.
COVID killed less than 1 per cent of humanity. The deadliest pandemic in recorded history was the Black Death, when the plague killed more than one-third of Europe’s population in the 13th century. However, even the plague would likely be much less deadly today due to our increased medical knowledge and better sanitation.
4. Nuclear war
What if AI got into the nuclear command and control chain and started a nuclear war? We’ve come close to nuclear war by mistake several times in the past 50 years.
We’re told that nuclear command and control is completely disconnected from the internet. But, as we saw in 2010, Iran’s nuclear centrifuges got taken out by a computer worm called Stuxnet, thought to have been brought in on a USB stick. AI can also give the military false intelligence, which could lead to irreparable actions.
The good news here is that nuclear stockpiles are down. But they are still enough perhaps to take out half of us. And it wouldn’t be by the nuclear blast itself, but the famine in the nuclear winter that would follow.
Imagine if AI causes massive job losses, pollutes the information space with misinformation, fractures our politics, and destroys human relationships with fake synthetic companionship
5. Other humans
Perhaps the most likely risk is that we take ourselves out. And AI might precipitate this.
Imagine – and it doesn’t take a lot of imagination – that AI causes massive job losses, pollutes the information space with misinformation, fractures our politics, and destroys human relationships with fake synthetic companionship.
Society might easily break. Slowly but surely, we’d stop being able to support human life at any scale.
What then to take away from all these scenarios? There are some things to be worried about for sure. But not to be too worried, I hope.
Toby Walsh is the author of God AI: boom or doom? What to expect when the machines outsmart us, published by La Trobe University Press.
– READ MORE: Trump unleashes on ‘globalist scheme’ championed by Anthony Albanese as the President launches bold bid to change name of artificial intelligence
