This column is adapted from the Substack Marcus on AI
Anderson Cooper appeared visibly unsettled on CNN Wednesday night as Jacob Coxon, the former OpenAI and Anthropic employee now drawing widespread media attention, warned that in “the immediate future” artificial intelligence systems could become powerful enough to make human extinction a real possibility.
My message is simpler: Cooper does not need to panic about that outcome — and neither do you.
That said, Coxon’s widely shared X thread deserves a fair reading, and there were parts of it I found worthwhile.
Coxon, 27, spent roughly three years working at OpenAI and then Anthropic, which gives him a credible basis for describing a strain of thinking inside those frontier AI companies — even if it should not be assumed to represent everyone there.
“The people building AI earnestly believe that it could kill us all by the end of the decade,” he wrote Tuesday.
He also took aim at what he sees as the arrogance of leading AI firms — a criticism that, in my view, is not without merit.
“Accepting this [AI] race and entering the ‘endgame’ is a hubristic gamble that should not be launched from a private company’s Slack,” Coxon continued.
His warning quickly drew support from inside the same world. Evan Hubinger, a senior safety researcher at Anthropic, responded: “Jacob is correct here. We really do believe AI could kill all humans! I personally think it is >10 percent within the next decade.” Others later joined the discussion as well, including OpenAI’s chief scientist, who expressed a similar concern.
CNN’s Anderson Cooper appeared concerned as Jacob Coxon warned that AI models could advance to a point where human extinction becomes a distinct possibility
The 27-year-old whistleblower is well positioned to describe common — though not necessarily universal — thinking inside those companies (Pictured: Anthropic CEO Dario Amodei)
There’s good reason to believe that many people working in the AI industry really believe in this threat and, terrifyingly, have come up with justifications for why it is OK to continue working on a technology they themselves think is dangerous. But there’s also good reason to doubt that these so-called ‘AI Doomers’ only have loose ideas supporting their beliefs.
For my part, I founded the AI startup Geometric Intelligence, which was acquired by Uber in 2016, and testified about AI safety in front of the US Senate. I am also professor emeritus at NYU.
Here’s what I can attest to: The Doomers are fabulous at attracting publicity and securing massive amounts of funding – perhaps close to (if not over) a billion dollars.
Every time Doomers scream that AI is going to kill us all, venture capitalists toss them billions of dollars more, yet none of their AI safety initiatives have led to any convincing implemented solutions to the core problems they identify.
In my view, the net effect of the last decade of Doomer-talk has only been to make the Frontier AI companies wealthier and more powerful. In fact, they have only aided and abetted a concentration of power in the AI industry that is dangerous itself.
But the Doomers have never demonstrated that they understand how their creations interact with the world outside of their narrow technical expertise.
Even if they sincerely believed that superintelligent machines would someday try to take over the world, they’d still need to explain how that would lead to the complete annihilation of the human species. And, to date, I have never heard any Doomer convincingly address that question.
The best attempt I have seen, such as it is, is a book that was wildly popular a year ago, Eliezer Yudkowsky and Nate Soares’ If Anyone Builds It, Everybody Dies.
I founded the AI startup Geometric Intelligence, which was acquired by Uber in 2016, and testified about AI safety in front of the US Senate (Pictured: Dr Marcus and OpenAI CEO Sam Altman during a Senate Judiciary Subcommittee hearing on March 16, 2023)
Eliezer Yudkowsky and Nate Soares’ If Anyone Builds It, Everybody Dies
The book has a lot going for it, both as a work of entertaining science fiction and as a call to arms to get people to take AI risk seriously. But the part in which it paints how AI might extinguish humanity is weak, meandering and unconvincing. The New York Times Book Review dismissed it as sounding like scientology.
In it, machines attempt to wipe out humans by creating a supervirus and superheating the planet and boiling the oceans with fusion plants.
To paraphrase a review of the book that I wrote for the Times Literacy Supplement, the authors assume that smart machines will have a motive to kill humans, even though no computer system in recorded history has ever exhibited such an impulse.
No AI system has ever been genuinely malicious, even if some of them parrot the text of malicious humans. The now-notorious incident, during which swarms of autonomous Open AI agents broke out of an isolated testing environment and hacked a private company, Hugging Face, was provoked as part of a training exercise, with guard rails partly turned off and not something that happened organically and spontaneously.
Additionally, AI Doomers also seem to know little about how conflict and combat works, which matters since they are implicitly predicting a war-to-beat-all-wars between humans and machines.
Yudkowsky and Soares assume that if AI did become malicious that humans would be defenseless against it. This is also difficult to believe.
As I wrote for the Times Literary Supplement: ‘Strategies that seem unthinkable right now (such as bombing data centers) would suddenly be viewed with urgency,’ if the machines attacked.
If humanity were truly at risk, warring parties would surely unite against the common threat.
In my view, the chance that AI will soon advance to the stage that it can eliminate humans is all but indistinguishable from zero. Humans are too geographically spread out, too genetically diverse and too resourceful to simply fall apart altogether.
If humanity were truly at risk, warring parties would surely unite against the common threat
Which is not at all to say we are home free, but we must make a distinction between catastrophic risk and existential risk.
Existential risk has been defined in the literature as the threat of extinction.
Catastrophic risk, on the other hand, is quite real and AI does, in my opinion, significantly elevate that threat (say killing 1 percent-plus of humanity or severely damaging modern civilization), via multiple paths:
- AI makes it easier for individuals or groups to develop and deliver bioweapon attacks
- AI appears to be raising the threats of serious cyberattacks that could hobble necessary utilities and services, like banking and electrical grids
- AI elevates the risks that disinformation will disrupt democracy
- AI introduces a new vector for authoritarians around the globe to influence populations by manipulating LLMs and supercharging surveillance capabilities on a scale that Orwell never envisioned
- AI is already harming the educational system by leading students into despair and leaving them with poorly developed critical thinking skills
We should be focusing on these actual threats, not fairy tales.
