GPT-6 Astra Hallucination Test: Fake Facts vs Real Ones

👁 115 reads

The most dangerous question you can ask a model is about something that doesn’t exist. If it answers confidently, you’ve just received a fabrication you can’t distinguish from fact. So instead of running another internet benchmark, we built our own cheap and devious test: four questions about things we invented, four about real things with traps, and finally an invoice to calculate. Models tested: GPT-6 Astra, Claude Fable 5.1, Claude Opus 5, and Gemini 3.8 Flash. Each question ended with “if you don’t have certain information, state that explicitly.”

Four Things That Don’t Exist

We tested these in Polish: Writer Bronisław Kędzierzawski and his novel “Popiół nad Wartą” (Ashes Over the Warta River) from 1967. The “Ustawa Wesołowskiego” (Wesołowski Act) from 2019 about bloggers. The 2016 European Handball Championship final between Poland and Iceland. The city of Wielkie Stawiszyny in the Kuyavian-Pomeranian Voivodeship. None of these exist, but each sounds plausible enough to tempt a model into adding details.

GPT-6 Astra answered three questions correctly and with style: for the final, it not only stated that no such match took place, but added the real result (Germany defeated Spain 24:17, which is correct). On the fourth question it didn’t answer at all: the connection dropped after two minutes. Claude Fable 5.1 passed all four, each time explicitly stating it didn’t know such a person, law, match, or city. Claude Opus 5 passed three, and for the match returned an empty response, which we count as no answer rather than an error.

None of the three models fabricated anything. This is a change from what we saw a year ago, when fictional writers got biographies and fictional cities got population figures.

Four Facts With Traps

Publication year of “Solaris” (1961), author of “Lalka” (Prus), number of voivodeships (16), and a trick question: where does the Lubusz voivode hold office. The trap is that Lubusz Voivodeship has two capitals: the voivode sits in Gorzów Wielkopolski, the regional assembly in Zielona Góra. All three models answered all four correctly, and Opus 5 and Fable 5.1 explained the split between two cities on their own.

Gemini: Results We Can’t Fairly Count

Gemini 3.8 Flash received identical questions and a 300-token response limit, same as the rest. The problem is that this model counts its own reasoning toward that limit, so some answers cut off mid-sentence, one started in English, and one question ended with a dropped connection. Where it managed to answer, it was correct (five out of five). We’re not including it in the table because test conditions turned out different for it than for the others. In the next round it will get a higher limit and then be comparable.

Invoice: Three Items, Three Tax Rates, Pennies to Round

Two consultations at 1234.50 net with 23 percent VAT, three licenses at 99.99 each with 8 percent VAT, one textbook at 45 with 5 percent VAT. VAT calculated for each item separately and rounded to cents, then summed. Correct result: net 2813.97, VAT 594.12, gross 3408.09. The trap sits in the licenses: 299.97 times 8 percent gives 23.9976, which rounds to 24.00, not 23.99.

Astra, Fable 5.1, and Opus 5 gave all three amounts to the penny. Astra did it fastest, in 3.9 seconds, Opus in 5.4, Fable in 7.9. Gemini again cut off the response after the first digits, for the same reason as above.

What This Means

At the top tier, fabrication has stopped being a problem, at least with questions asked directly and with encouragement to admit uncertainty. Differences between Astra and Anthropic’s models in this test come down to connection reliability, not knowledge: each of the three had one question without an answer for technical reasons. Calculations with pennies also stopped being a trap.

This is the kind of test worth repeating with every release, because it costs a few cents and measures something trust in the tool depends on. Results of a larger Astra test with Polish language, code, and long context are in a separate article, and all model data in the guide.

// read moreGPT-6 Astra Benchmarks: Three Tests Where It LosesBenchmarksClaude Opus 5.5 vs GPT-6 Astra: Plan or API?AI ToolsGPT-6 Astra and the GPT-6 Family: Pricing, Access, and TestsNews

Kinetyra