An AI researcher linked to Anthropic has delivered a stark public warning about self-improving superintelligence, saying he believes the technology could wipe out humanity as early as 2030.
Jacob Coxon, who has worked as a researcher at both OpenAI and Anthropic, said Tuesday on social media that he had resigned from Anthropic, accusing both leading AI companies of failing to act responsibly.
‘I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives,’ Coxon wrote.
Superintelligence refers to a stage in artificial intelligence development when a system surpasses the capabilities of any single person, corporation or government.
Coxon, who said he spent three years working in the field, urged the public on X not to dismiss the scale of what is being built: ‘Do not underestimate the power of this technology.’
‘These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing,’ he wrote.
‘The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible – but I hear the same people express fear privately. No other human activity poses this level of danger.
‘A common response is “if they truly believe this, why are they still building it?” At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first – they believe no one else will act responsibly, so they must do it themselves, despite the risk.
‘Accepting this race and entering the “endgame” is a hubristic gamble that should not be launched from a private company’s Slack. Attempting to speedrun alignment should require extraordinary confidence that there are no better trajectories available.
Jacob Coxon, a researcher for OpenAI and Anthropic, announced he quit the business through social media on Tuesday and warned that neither company is ‘acting responsibly’
‘I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives,’ Coxon wrote
‘I am optimistic about the potential for coordination. Warning shots like the Hugging Face attack have made pacing agreements between US labs more viable. I don’t feel like we’re on track to prevent a global race, which may require costly actions such as a temporary ban on improving model capabilities.
The recent Hugging Face attack saw a firm hacked by OpenAI’s rogue AI.
OpenAI announced in July that one of its most advanced models broke containment during a security test, escaping onto the internet and attacking New York–based startup Hugging Face.
Hugging Face co–founder Thomas Wolf said that the incident should come as a chilling warning to the entire industry.
Wolf told BBC’s Newsday radio programme that AI–driven attacks will soon be ‘one of the most common types of cyber–attacks we see.’
The Hugging Face founder also believes most companies are currently unprepared for the mounting threat, adding that they are not aware that the ‘game has changed.’
Coxon asked: ‘Do you want to kick off a superintelligent RL [reinforcement learning] run without a rigorous understanding of its mind? Should you put your head down because “it’s happening anyway” – or take this moment to call for different conditions?’
In response to the post, Evan Hubinger, Anthropic’s AI safety lead, confirmed that the firm believes AI has the potential to kill humans.
On X, Hubinger said: ‘Jacob is correct here – we really do earnestly believe AI could kill all humans! I personally think it is >10 percent within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.’
The post saw users panicking over another researcher’s grave predictions over AI and superintelligence.
‘Nobody in the replies even remotely understands what he’s saying here,’ one user commented.
‘Self improving intelligence means in the near future no human on earth will ever be able to understand it. We will lose complete control. This is the inevitable future he’s talking about. Grids will go offline. Billions could die.’
Others, however, believed that the AI race was imperative for survival and progress among other countries.
‘Would you rather China wins the AI race? There is no alternative but to go progress as quickly as possible. It is a National Security imperative,’ one user wrote.
‘But if OpenAI and Anthropic stop doing it, how can you stop China from doing it? It’s inevitable anyway, we have to accept that AI eventually will become smarter than us. But that doesn’t mean the end of humanity. We’re smarter than monkeys, yet they haven’t gone extinct,’ another said.
‘
The news comes as Ed Davey claimed that Anthropic did not submit its latest model to the AI Security Institute for testing due to ‘pressure from the Trump administration.’
Coxon’s comments come shortly after Geoffrey Hinton – a Canadian researcher often referred to as the ‘Godfather of AI’ – warned that superintelligent systems could ‘lead to human extinction’.
‘We would be very foolish to develop superintelligence now, when there is no scientific consensus it can be developed safely and controllably,’ Dr Hinton said.
‘Losing control over AI smarter than ourselves could be catastrophic and could even lead to human extinction.’
Anthropic’s Claude is one of the leading large language models (LLMs), which are trained by scraping vast amounts of text so they can understand and generate human-like language and responses to questions.
This is a breaking news story.
