Why AI Can’t Tell If You’re Making a Vaccine or a Weapon

👁 115 reads

In Anthropic’s report describing misuse of its model, one sentence matters more than all the case descriptions: the same information that enables someone to develop a biological weapon also enables them to develop a vaccine. This isn’t a corporate excuse - it’s a description of a problem no one has solved yet, and it affects everyone who uses AI for anything serious.

What Dual-Use Means

Dual-use is a property of knowledge, not of human intention. The same technology, the same formula, the same procedure serves both good and harm, depending solely on who reaches for it.

The concept predates artificial intelligence by a good eighty years. Nuclear fission gives us electricity and bombs. Nitrogen fertilizers fed billions of people and form the basis of explosives. Research on how viruses jump between species lets you prepare a vaccine before an epidemic breaks out - and is exactly the knowledge someone with opposite intentions would need.

What’s new is that for the first time, this knowledge is being dispensed by an automaton that has no way to verify who’s on the other side.

Why Models Can’t Recognize Intent

A human in a laboratory has context. They know who’s asking, what institution they work for, what they did last month, and whether the question fits their project. A model has none of these things. It sees a string of characters in a chat window.

Anthropic describes this without hedging: they want their models to be useful in scientific research, and they expect AI to transform biology and accelerate drug development. In a simple world, malicious uses would be obvious because someone would explicitly declare their intent. But, as they write, we don’t live in that world.

Worse: the company notes that sophisticated actors know about this ambiguity and deliberately exploit it to maintain the appearance of ordinary research. In one described case, a researcher allegedly asked the model to deliberately describe factors imprecisely in progress reports.

This closes off a certain line of thinking. You can’t build a filter that distinguishes malicious intent from question content if the questioner knows such a filter exists and formulates their question accordingly.

The Price We All Pay

A safeguard that can’t recognize intent has only one option: block more broadly than necessary. Anthropic stated this explicitly, explaining that they released their latest models with restrictions covering a wide range of questions about dual-use biological research.

The word “wide” is key here and means exactly what people with no connection to weapons complain about. A student asking about the mechanism of botulinum toxin, a paramedic asking about dosages, a journalist writing about an epidemic, and a teacher preparing a lesson all hit the same filter.

I don’t know any way to make this cost disappear. You can shift it, but not eliminate it: either you let through more dangerous questions, or you block more innocent ones. No one has shown a third option yet.

Why One Company’s Caution Achieves Nothing

The report contains a detail that invalidates most discussions about manufacturer responsibility. When the model refused to answer, the same queries went to competitors’ models with looser safeguards.

If a blocked question can simply be asked elsewhere, then the most cautious provider’s safeguard stops being a barrier and becomes an inconvenience. It costs their users but doesn’t stop anyone.

That’s why Anthropic writes that they’re publishing this report partly so other providers can recognize similar patterns in their own systems. This is an admission that this problem can’t be solved alone, and simultaneously an admission that today no one has the tools to force the rest of the industry to the same level of caution.

What This Doesn’t Mean

It doesn’t mean models are dangerous and need to be shut down. The same capability that raises concerns in the report is the one that enables new drugs to be developed.

It also doesn’t mean safeguards are useless. Anthropic detected these cases, blocked accounts, and passed findings along. The system worked, just after the fact and not in every case.

And it doesn’t mean anyone built a weapon. The report describes work that could support such a goal, and the company explicitly states they’re not claiming the described individuals had malicious intent.

What This Means Practically

When a model refuses nonsensically, it’s usually not a malfunction but a safety margin. It’s worth rephrasing your question and describing your goal rather than looking for a workaround. For educational questions, this works more often than you’d expect.

For professional work, check this in advance. If you’re building something on a model and your field touches medicine, chemistry, or security, test exactly where that model stops before you base a process on it. Discovering this mid-implementation is far more expensive.

Don’t assume all models block the same things. The report shows that differences between providers are large enough that they became part of someone’s strategy for circumventing safeguards.

We covered the details of the report itself, including what Anthropic disclosed and what they deliberately withheld, in a separate article. For how competing OpenAI’s chief scientist views the same issue, see our piece on his essay about slowdown.

// read moreAnthropic Bans Cruelty and Abuse Toward AI in New PolicyNewsDario Amodei: Anthropic CEO Net Worth, Age & AI SlowdownMagazineAnthropic: The Company Behind Claude AIMagazine

Kinetyra