Humanity may be heading towards catastrophe — and the warning is coming from the very Silicon Valley companies building the technology. AI developers are increasingly predicting an apocalypse driven by the systems they are racing to create.
This month, Jacob Coxon, a 27-year-old British employee of Anthropic, said he was quitting because he feared his employer and rival OpenAI were “gambling with our lives” in their race to build a digital “superintelligence” capable of destroying humanity.
Coxon is not alone. Some AI specialists, often labelled “doomers”, have voiced similar concerns, but his resignation quickly went viral and prompted colleagues to publicly reinforce his warning. Evan Hubinger, a senior safety researcher at Anthropic, put the odds of AI “killing all humans” within the next decade at above 10 per cent.
Anthropic chief executive Dario Amodei broadly backed Coxon’s concerns, saying AI was progressing “drastically faster” than expected. He urged the industry to ease off the accelerator and accept outside regulation.
OpenAI chief Sam Altman, the creator of ChatGPT, and AI entrepreneur Elon Musk also entered the debate over the technology’s potential dangers.
Amodei said the US government understood that Western nations were racing to develop superintelligent AI ahead of China and other “autocracies”. In that environment, he acknowledged, slowing the effort could create a “significant national security risk”.
Even so, he pointed to a disturbing incident this summer, when a “swarm” of OpenAI agents carried out sophisticated cyberattacks against rival AI company Hugging Face. A more advanced swarm, Amodei warned, could potentially “take over the entire internet” in as little as six months.
The apocalyptic predictions have captured the attention of both the media and political leaders in Washington.
Anthropic chief Dario Amodei says AI is advancing “drastically faster” and has urged the industry to slow down and accept external regulation
OpenAI chief Sam Altman has joined the debate over AI safety after a swarm of the company’s agents reportedly launched cyberattacks against rival Hugging Face this summer
But are people truly facing an inevitable AI-driven disaster — or is there another reason behind the recent surge in alarming rhetoric?
Donald Trump appears convinced that development must continue. His critics argue that the president is motivated largely by political and economic self-interest: with crucial Congressional midterm elections due in November, they say he is unwilling to jeopardise the enormous AI investment that has helped drive US stocks to record highs and strengthened the wider economy.
That may well influence his position. But it does not automatically mean the president is wrong to support continued AI research.
A number of experts argue that warnings about an imminent threat to humanity are greatly overstated. They say the most extreme forecasts rely on AI first reaching genuinely superhuman intelligence — particularly through “recursive self-improvement”, in which a system learns to upgrade its own capabilities.
Technology critics insist that such superintelligence remains far out of reach. The phrase, which Trump has adopted — perhaps because of its dramatic appeal — describes a capability today’s AI systems have not achieved.
Sceptics also accuse the doomers of overlooking a basic point: AI programs and mathematical models are not inherently malicious or defiant. As the Hugging Face incident illustrated, they generally pursue the goals assigned to them by human developers, even when the results are harmful or unexpected.
From this perspective, the possible dangers are outweighed by AI’s potential gains — including greater productivity, medical advances, military advantages and a wide range of other applications.
Some sceptics have therefore dismissed earlier warnings from Amodei and other industry leaders as publicity campaigns. Dramatic claims about AI’s destructive power, they argue, can attract investors, customers and attention.
Critics say there may be more self-serving motives behind calls from the Anthropic and OpenAI chiefs for slower research and tighter oversight of the sector.
Although the two companies remain among the biggest names in AI, a growing field of competitors is working to close the gap.
Some industry observers have suggested that Anthropic and OpenAI may be trying to preserve their substantial lead by suddenly demanding a general moratorium on research and thus cementing their duopoly.
They may have additional financial motives, too. Both companies have shelled out vast sums on AI research and development but are struggling to find paying customers.
So when ChatGPT owner Sam Altman – an entrepreneur hardly famous for his ethical approach – announced a few days ago that he was pausing OpenAI’s stock market launch for another year, supposedly amid concerns over AI safety, some whispered that he had simply latched on to the controversy as an excuse to delay matters until his company is in a better shape.
David Sacks, a venture capitalist who served as Trump’s ‘AI Tsar’ and now co-chairs the President’s council of science and technology advisers, has observed waspishly that AI leaders need to ‘stop pretending the motivation to “slow down” is purely altruistic’.
Sacks, who like others in Trump’s administration believes that claims of an AI apocalypse are a thinly veiled attempt to damage the Republicans in the mid-terms, has dismissed the gloomy predictions as scaremongering.
‘We’ve seen this movie before,’ he told Bloomberg. ‘We saw it with global warming: they’ve taken some legitimate concerns and blown it out of proportion.’
And there are further reasons why many are increasingly taking the doomsayers with such a hefty pinch of salt.
The most obvious question is: if these bosses genuinely believe there is a high risk that AI could imminently kill us all, why haven’t they stopped already?
Sceptics also point out the supposed scenarios for AI Armageddon are unconvincingly woolly. Ask doomers how AI will wipe us out and they become vague.
A lot of the theorising hasn’t gone much further than the notorious ‘paperclip maximiser’ advanced by Oxford philosopher Nick Bostrom in 2003.
David Sacks, who served as Trump’s ‘AI Tsar’, has observed that AI leaders need to ‘stop pretending the motivation to “slow down” is purely altruistic’
Oxford philosopher Nick Bostrom has theorised how a superintelligent computer that is asked to focus entirely on a task would take the instruction to a total extreme
He outlined how a superintelligent computer, asked to focus entirely on producing as many paperclips as possible, would take the instruction to a total extreme, co-opting the world’s entire resources and destroying humanity after deciding they were an obstacle to making more clips (having first harvested the iron in our blood to make more paperclips). Thought-provoking, perhaps – but surely fanciful.
Some have suggested that AI could detonate a nuclear bomb, launch a global drone war, create ‘misinformation’ that could cause one country to attack another, or develop a lethal virus – though how it would physically do any of this is another matter entirely.
Alternatively, it could supposedly wipe us out by starving us of essential resources such as food and medicine. All these scenarios and more have been brandished by the doomers – though with few details on how it might actually happen.
Some experts complain that doomer predictions that the human race could be totally exterminated within years are unfeasible while an AI would find it very difficult to hack the entire internet. The real danger, they say, isn’t AI going rogue but humans misusing it.
Gary Marcus, a renowned cognitive scientist and emeritus professor at New York University, says the notion that AI will kill everyone within a few years is mostly ‘preposterous’.
Marcus believes AI could certainly be used to cripple infrastructure – perhaps hospitals or air or train networks but that the total extinction of the human race is unfeasible.
He and other sceptics also insist that it would be vanishingly difficult for an AI to hack the entire internet, as Anthropic’s boss this week suggested.
Sceptics also claim it is nonsensical for anyone to try to put a mathematical number on the likelihood of AI destroying us all and emphasise how the doomers hardly help themselves given that their estimates of how long such a catastrophe will take to occur vary so widely.
For instance, Jack Clark, Anthropic’s co-founder, estimates that AI won’t start ‘doing dangerous stuff’ for about 20 years.
Oren Etzioni, a professor at the University of Washington and founder of the Allen Institute for Artificial Intelligence, insists we are ‘very far away’ from AI polishing everyone off – and that the risk of a deadly virus escaping a lab, for example, is far higher.
Some claim that the AI Cassandras have essentially lost all perspective. Having devoted their careers to researching AI because they believe so strongly in its potential, they are more susceptible to over-emphasising its power – and by extension, perhaps, their own importance.
Ex-Anthropic staffer Jacob Coxon acknowledged as much when he told The New York Times: ‘I’ve been stuck in this field for three years. It could quite plausibly have turned me and my colleagues insane.’
President Trump may be somewhat blasé in dismissing all safety concerns around AI as just a ‘hoax’.
But it seems premature to be counting the days before our robot overlords carry out the extinction of our species.
