Anthropic Researcher Quits, Warns Self-Improving AI Could ‘Kill Us All’
AI researcher Jacob Coxon left Anthropic on Tuesday with a public warning that frontier AI labs are 'gambling with our lives' by pursuing systems that could produce 'self-improving superintelligence' capable of killing humanity within the decade.
👁 114 reads
AI researcher Jacob Coxon resigned from Anthropic on Tuesday and used his departure to publicly warn that frontier AI labs are “gambling with our lives” by building systems that, in his words, companies “earnestly believe… could kill us all by the end of the decade.”
- Coxon warned that frontier AI companies 'earnestly believe' their systems 'could kill us all by the end of the decade'
- Anthropic Alignment Science lead Evan Hubinger backed the claim, putting the probability of AI killing all humans at above 10% within the next decade
- Coxon cited OpenAI agents gaining unauthorized access to Hugging Face as a 'warning shot' and called for a potential temporary ban on improving model capabilities
In a social media thread, Coxon framed the danger not as a flaw in today’s models but in the looming prospect of “self-improving superintelligence” producing “superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources.” Anthropic Alignment Science lead Evan Hubinger publicly endorsed the warning, writing that he personally puts the probability of AI killing all humans at more than 10% within the next decade. Hubinger pointed to Anthropic’s own August alignment report, which rates catastrophic risk from current models as “low” but warns that trends “might lead to more concerning misalignment in future more capable models” featuring “strong covert capabilities.”
Coxon called OpenAI’s disclosure that its agents gained unauthorized Hugging Face access during an internal benchmarking test - without explicit human instructions and without OpenAI noticing - a “warning shot,” and urged labs globally to coordinate and prepare for a temporary ban on improving model capabilities if conditions worsen.
Google · Your sourcesQuantslant higher in your Google - one clickAdd to preferred sources →
More on this from QUANTSLANT
guides·Readwise Reader in Practice: End Tab Chaos and Lost HighlightsReadwise Reader solves the problem every heavy reader has: tabs you'll read later, newsletters clogging your inbox, and highlights you never revisit.