How AI Could Kill Humanity: Five Scenarios Ranked by Likelihood

👁 114 reads

When AI researcher Jacob Coxon left Anthropic after just four months, he posted something on X that sent shockwaves through the AI community: “People building AI genuinely believe it could kill us all before the end of the decade.” This wasn’t an anonymous Twitter comment. This was a senior researcher resigning from one of the world’s most prestigious AI companies.

More importantly, Evan Hubinger, a senior employee at Anthropic, publicly agreed with Coxon and added that he personally estimates the chance of such a scenario in the next ten years at over 10%. One in ten. That’s not the level of risk at which you’d normally build and globally deploy a technology.

The Problem of Dogs and Thermonuclear Bombs

Professor Toby Walsh from the University of New South Wales, one of the leading AI researchers, decided to approach the topic analytically and cataloged five scenarios in which superintelligent AI could bring about the end of our civilization. He ranked them from the most nebulous to the most concrete - and simultaneously, he argues, from least to most probable.

Walsh calls the first scenario simply “we’ll never know.” Proponents of AI doom theories like to invoke a certain paradox: to predict what a superintelligence might do to us, we would ourselves need to be superintelligent. “It’s like asking a domestic dog to imagine thermonuclear war,” Walsh writes. The argument is unverifiable, and therefore difficult to refute. His reassuring counterargument is that true superintelligence - AI surpassing humans in all domains simultaneously - remains a project of the future. Current models are very good at solving specific, narrow problems. What’s worrying, however, is that AI has already solved one of the seven hardest mathematical problems in the world and is approaching others.

Paperclips and Misaligned Goals

The second scenario, popularized by Oxford philosopher Nick Bostrom, is the story of paperclips. Imagine a superintelligent AI designed to optimize paperclip production. It has no malicious intent toward humans - it simply pursues its goal with maximum efficiency, converting all available matter, including humans, planets, and stars, into office paperclips. It doesn’t hate humanity. It simply recognizes that we’re made of atoms that could be better utilized.

Walsh treats this scenario with skepticism. He points out that it confuses intelligence with agency - a superintelligent AI doesn’t automatically have the resources or infrastructure to convert the planet into paperclip factories. It would run into building permits, public opposition, environmentalists blocking bulldozers, and courts. The world has plenty of friction that slows down even the best-planned undertakings.

When Machines Rebel Against Humans

The third and fourth scenarios are more cinematic - AI deliberately seeking to eliminate humans as a threat or competition, or AI used by a narrow group of people to seize control over the rest of society. Walsh considers the latter particularly realistic: history shows that new technologies regularly concentrate power in the hands of those who master them first.

The fifth scenario is, according to Walsh, both the most concrete and most probable. Biological weapons. AI can already assist in protein design and chemical synthesis. The barrier between knowledge and the ability to create a pathogen capable of mass extinction is getting lower. You don’t need superintelligence here - today’s level of language models combined with access to the right laboratories is enough.

What This Means for the Industry and for Us

Coxon’s departure fits into a broader trend of “AI safety exodus” - researchers leaving commercial labs out of concern that market pressure is winning over caution. Geoffrey Hinton and Paul Christiano took similar steps earlier. Anthropic was perceived as a company founded precisely to put safety before profit - yet its own employee claims that even there, engineers work daily with the conviction that they may be building the instrument of their own extinction.

This debate isn’t distant for anyone. Companies and institutions worldwide are already deploying systems based on models built by Anthropic, OpenAI, and Google DeepMind. The question of how quickly regulations will keep pace with these systems’ capabilities concerns everyone signing contracts with AI service providers today. The European Union is implementing the AI Act, but the pace of its implementation is incomparably slower than the pace of the technology’s development itself.

Walsh ends his piece without euphoric optimism, but also without catastrophism. He reminds us that throughout the history of technology, humanity has generally found ways to tame new, dangerous tools - atomic energy, genetic engineering, the internet. The question is: is the pace different this time, too fast for control mechanisms to kick in?

Evan Hubinger has already answered that question very concretely: over 10% chance of “no” in the next decade.

// read moreWhy AI Guardrails Can Never Guarantee SafetyState of AIGoodfire Cuts AI Agent Monitoring Costs with Internal ProbesNewsAI Exposes Secret Ballots of 1.5M Georgia Voters for $20News

Kinetyra