This column is adapted from the Substack newsletter Marcus on AI
Anderson Cooper appeared visibly uneasy on CNN on Wednesday night after Jacob Coxon, a former OpenAI and Anthropic employee who has quickly become a media focal point, warned that artificial intelligence could soon reach a stage where human extinction is a genuine possibility.
Here is the reassurance Cooper—and everyone else—may need: that particular scenario is not something we should be losing sleep over.
Before explaining why, however, it is worth acknowledging what Coxon got right in his now widely shared post on X.
The 27-year-old whistleblower spent roughly three years at OpenAI and, more recently, Anthropic. That experience gives him a credible view of the prevailing—though presumably not universal—attitudes inside those companies.
“The people building AI earnestly believe that it could kill us all by the end of the decade,” Coxon wrote Tuesday.
He also criticized the leading AI developers for what he described, rightly in my view, as extraordinary arrogance.
“Accepting this [AI] race and entering the ‘endgame’ is a hubristic gamble that should not be launched from a private company’s Slack,” he wrote.
Coxon’s claims soon received support from Evan Hubinger, a prominent AI-safety researcher at Anthropic. “Jacob is correct here,” Hubinger wrote. “We really do believe AI could kill all humans! I personally think it is >10 percent within the next decade.” Others later expressed similar views, including OpenAI’s chief scientist.
CNN anchor Anderson Cooper appeared visibly concerned after Jacob Coxon warned that AI models could advance to the point where human extinction becomes a real possibility.
The 27-year-old whistleblower’s time at OpenAI and Anthropic gives him a credible perspective on the views commonly held—though presumably not by everyone—inside those companies. Above, Anthropic CEO Dario Amodei.
AI ‘Doomers’ Accused of Profiting From Extinction Fears
There is ample reason to think that many AI professionals genuinely fear the technology could pose an existential threat. More troubling still, some have developed arguments for continuing to build a system they believe may be dangerously uncontrollable. Yet there is also reason to question whether the so-called “AI Doomers” have more than loosely connected ideas to support those sweeping conclusions.
For my part, I founded the AI startup Geometric Intelligence, which was acquired by Uber in 2016, and testified about AI safety in front of the US Senate. I am also professor emeritus at NYU.
Here’s what I can attest to: The Doomers are fabulous at attracting publicity and securing massive amounts of funding – perhaps close to (if not over) a billion dollars.
Every time Doomers scream that AI is going to kill us all, venture capitalists toss them billions of dollars more, yet none of their AI safety initiatives have led to any convincing implemented solutions to the core problems they identify.
In my view, the net effect of the last decade of Doomer-talk has only been to make the Frontier AI companies wealthier and more powerful. In fact, they have only aided and abetted a concentration of power in the AI industry that is dangerous itself.
But the Doomers have never demonstrated that they understand how their creations interact with the world outside of their narrow technical expertise.
Even if they sincerely believed that superintelligent machines would someday try to take over the world, they’d still need to explain how that would lead to the complete annihilation of the human species. And, to date, I have never heard any Doomer convincingly address that question.
The best attempt I have seen, such as it is, is a book that was wildly popular a year ago, Eliezer Yudkowsky and Nate Soares’ If Anyone Builds It, Everybody Dies.
I founded the AI startup Geometric Intelligence, which was acquired by Uber in 2016, and testified about AI safety in front of the US Senate (Pictured: Dr Marcus and OpenAI CEO Sam Altman during a Senate Judiciary Subcommittee hearing on March 16, 2023).
AI Extinction Scenario Branded ‘Weak and Unconvincing’
The book has a lot going for it, both as a work of entertaining science fiction and as a call to arms to get people to take AI risk seriously. But the part in which it paints how AI might extinguish humanity is weak, meandering and unconvincing. The New York Times Book Review dismissed it as sounding like scientology.
In it, machines attempt to wipe out humans by creating a supervirus and superheating the planet and boiling the oceans with fusion plants.
To paraphrase a review of the book that I wrote for the Times Literacy Supplement, the authors assume that smart machines will have a motive to kill humans, even though no computer system in recorded history has ever exhibited such an impulse.
No AI system has ever been genuinely malicious, even if some of them parrot the text of malicious humans. The now-notorious incident, during which swarms of autonomous OpenAI agents broke out of an isolated testing environment and hacked a private company, Hugging Face, was provoked as part of a training exercise, with guardrails partly turned off and not something that happened organically and spontaneously.
Additionally, AI Doomers also seem to know little about how conflict and combat works, which matters since they are implicitly predicting a war-to-beat-all-wars between humans and machines.
Yudkowsky and Soares assume that if AI did become malicious that humans would be defenseless against it. This is also difficult to believe.
As I wrote for the Times Literary Supplement: ‘Strategies that seem unthinkable right now (such as bombing data centers) would suddenly be viewed with urgency,’ if the machines attacked.
If humanity were truly at risk, warring parties would surely unite against the common threat.
In my view, the chance that AI will soon advance to the stage that it can eliminate humans is all but indistinguishable from zero. Humans are too geographically spread out, too genetically diverse and too resourceful to simply fall apart altogether.
If humanity were truly at risk, warring parties would surely unite against the common threat.
AI Poses Catastrophic Rather Than Existential Risks
Which is not at all to say we are home free, but we must make a distinction between catastrophic risk and existential risk.
Existential risk has been defined in the literature as the threat of extinction.
Catastrophic risk, on the other hand, is quite real and AI does, in my opinion, significantly elevate that threat (say killing 1 percent-plus of humanity or severely damaging modern civilization), via multiple paths:
- AI makes it easier for individuals or groups to develop and deliver bioweapon attacks
- AI appears to be raising the threats of serious cyberattacks that could hobble necessary utilities and services, like banking and electrical grids
- AI elevates the risks that disinformation will disrupt democracy
- AI introduces a new vector for authoritarians around the globe to influence populations by manipulating LLMs and supercharging surveillance capabilities on a scale that Orwell never envisioned
- AI is already harming the educational system by leading students into despair and leaving them with poorly developed critical thinking skills
We should be focusing on these actual threats, not fairy tales.
