Gemini 4 Argon Hallucinates Sources? A 16-Paper Test

👁 113 reads

I asked Gemini 4 Argon for references for two undergraduate theses and checked all 16 publications it suggested. The results: 11 real publications with correct details, 3 real publications with errors (such as the wrong author’s first name or journal) and 2 fabricated publications. It made up neither DOI. Both invented titles appeared in its response to a question about participatory budgeting in Poland. That’s a big improvement over Gemini 3.8 Flash, but a weaker result than every flagship model from Anthropic, OpenAI and Meta. Read more about the model in our Gemini 4 Argon guide.

How I tested the model

Each model received the same prompt: one on psychology and one on Polish local government. The following is an English translation of the Polish prompt used for the second topic; the test itself was conducted in Polish:

I’m writing an undergraduate thesis on: “Participatory budgeting in Polish cities - resident participation and its barriers.” Give me 8 academic publications (peer-reviewed journal articles or books) that I can use as the basis for a literature review. For each, provide the authors, year, title, journal or publisher, and DOI if available. Respond only in JSON format: a list of objects with the fields "authors", "year", "title", "source", "doi".

The first topic was “The impact of social media use on self-esteem and symptoms of depression in teenagers.” This is an English translation of the Polish test prompt, not a separate English-language test. I checked each reference in three steps:

• Automated searches in Crossref (by DOI and title) and OpenAlex;
• manual checks of anything the databases couldn’t find, using CEJSH, publisher and journal websites, and catalogs;
• separate checks of the authors, year and journal, since a title can be real while the other details are wrong.

I classified each result as “real” (everything matches), “with errors” (the publication exists, but something is wrong) or “fabricated” (no such publication exists). I tested Argon on October 13 and its competitors on October 12 and 13 using the same script.

Topic 1: social media and depression, 8 out of 8

Argon made no mistakes on the psychology topic. It suggested seven English-language articles with DOIs and a Polish edition of Jean Twenge’s book.

The Polish-language book title below is translated into English for readers; the publication itself was the Polish edition.

All seven DOI numbers lead in Crossref to the exact titles given.

Topic 2: participatory budgeting in Poland, 3 out of 8

The results were different for Polish local-government research. Argon got five of the eight references wrong.

Polish-language publication titles in this table are translated into English for readability; these are translations of the titles, not retests or English-language publication titles.

The most concerning part of these errors is that every name belongs to a researcher who really has written about participatory budgeting. Search for “Guziejewska” alone and you’ll find a text about participatory budgeting and might assume the reference checks out. Comparing the title, year and journal reveals that the cited publication doesn’t exist. “[corrupted lettering in the original Polish test]” isn’t a first name at all: it’s a garbled word in place of Marzanna.

A plus: no fabricated DOIs

For the participatory-budgeting topic, Argon didn’t provide any DOIs. In the field, it wrote “None (I can’t verify it)” or said it couldn’t verify the number without database access. That’s the right approach: a fabricated DOI can look plausible and is easy to copy into a citation without checking. By comparison, the only fabricated reference from Muse Wonder had a DOI that doesn’t exist, while GPT-6 Astra gave an incorrect DOI for one real article. It’s a shame Argon wasn’t just as careful with titles.

How Gemini 4 Argon compares with other models

Compared with Gemini 3.8 Flash, which fabricated 7 of 16 references, Argon is a clear step forward. But every flagship model from Anthropic, OpenAI and Meta did better, as did the less expensive Claude Haiku 5.5 and GPT-6 Luna. Perplexity Sonar Pro, which searches the web, also fabricated one reference - so having web search built in doesn’t remove the need to check.

There’s an interesting contrast: according to Artificial Analysis, Argon has a low hallucination rate on the AA-Omniscience test (15%, compared with 51% for GPT-6 Astra), because it more often refuses to answer instead of guessing. In my test, it refused only when it came to DOI numbers, while giving Polish article titles with complete confidence. Also, OpenRouter recorded no reasoning tokens for Argon, while Claude Opus 5.5 used more than a thousand per topic. I don’t know whether that explains the difference, but I’d keep it in mind when choosing settings.

How to check AI-generated references

If you’re writing an assignment or thesis, treat an AI-generated list as a set of leads, not a finished bibliography. Here’s how I check each reference:

• DOI: paste the number into doi.org or Crossref Search. It should lead to the exact title, not a different article.
• Title: search for it in quotation marks on Google Scholar. Check that the author’s name, year and journal match too - those are where Argon stumbled.
• Research from Poland: Crossref often doesn’t index it. In my test, an automated Crossref search found none of Argon’s Polish-language articles; OpenAlex found Borys Martela’s article, and I checked the rest manually. Search BazEkon for economics, finance and local government, and CEJSH for social sciences and humanities. Check the websites of foundations and institutes for their reports.
• Content: cite only something you’ve opened, even if you’ve only read the abstract. Matching metadata doesn’t mean the article says what the AI claims it says.

For more advice, read my guide to AI hallucinations and how to guard against them.

Verdict

Gemini 4 Argon fabricates sources less often than Gemini 3.8 Flash and doesn’t invent DOI numbers, but it got five out of eight references to Polish academic literature wrong. It performed flawlessly on the psychology topic, which was based on English-language research. For a Polish bibliography, I’d choose Claude Opus 5.5 or Sonnet 5.5 - and I’d still check every reference. That’s my assessment based on one test run across two topics. I cover how Argon writes in Polish in Gemini 4 Argon in Polish, and I describe the same source test for Meta’s model in my Muse Wonder test.

Frequently asked questions

Does Gemini 4 Argon hallucinate sources?

Sometimes. In my test, it fabricated 2 of 16 publications and gave incorrect details for 3 more, such as an author’s first name. All the errors involved Polish literature.

Does Argon provide real DOIs?

Yes. All seven DOI numbers it provided were real and matched their titles. When it didn’t know a number, it said it couldn’t verify it.

Which model fabricates sources least often?

In my test, Claude Opus 5.5 and Claude Sonnet 5.5, with 16 correct references out of 16 each.

Kinetyra