AI Model Refuses to Read Documents: Testing Three Providers
A context test revealed content_filter triggers for every document above ~2000 tokens across AWS, the producer, and Google - even for text about plants.
👁 117 reads
I prepared an AI model context test: a document with hidden information, questions about it, and a check of how much the model could find. The result was zero, so I briefly assumed the model had poor memory. After examining the raw API response, it turned out the model hadn’t read a single word - the input was rejected by moderation and returned with content_filter and native_finish_reason: refusal.
Key Finding in One Paragraph
Zero hits in a memory test isn’t always proof of short context. In my measurement, the model returned content_filter and native_finish_reason: refusal for EVERY document above approximately 2000 tokens, and this happened across three independent providers - AWS, the producer’s API, and Google. The refusal applied even to text about plants, which contained no technical content, tool names, or anything that could be considered sensitive. When I substituted an alternative model on the same material and the same questions, it hit 6 out of 6 hidden pieces of information in 2.3-4.9 seconds. So the task was achievable, and the earlier result measured moderation, not memory.
How the AI Context Test Worked
The scheme was simple and deliberately boring. I took a document, wove six specific pieces of information into it at different locations - at the beginning, in the middle, near the end - and then asked six questions, one for each hidden piece of information. I counted hits and response time. Nothing more.
I started the test material with a short version and scaled upward. This is important because this procedure allowed me to see where the boundary lies. Below approximately 2000 tokens, responses came normally. Above that - content_filter, every time, without exception.
To rule out content as the reason, I swapped the document for text about plants. Zero technical terminology, zero code, zero anything controversial. Identical refusal.
Three Providers, Same AI Context Limit
The first thought when facing refusal is usually: something in the configuration of this specific endpoint. That’s why I repeated the test across three independent access paths - AWS, directly at the model producer, and Google. These are separate infrastructures, separate panels, separate security settings. The behavior was identical.
This is the strongest element of the entire measurement for me. If the problem resided in my code or in one incorrectly set moderation threshold, it wouldn’t have repeated across three platforms simultaneously.
| Element | Before Diagnosis (Refusing Model) | After Substitution (Alternative Model) |
|---|---|---|
| Hits on Hidden Information | 0 - no substantive response | 6 out of 6 |
| Response Time | not applicable - refusal | 2.3-4.9 s |
| Returned Status | content_filter, native_finish_reason: refusal |
normal completion |
| Refusal Thresholds | every document above ~2000 tokens | none |
| Text About Plants | refusal | processed |
| Number of Providers with This Behavior | 3 out of 3 (AWS, producer, Google) | 0 |
Why “Model Refuses to Respond” Is Easy to Confuse with Poor Memory
Most client libraries and wrappers show empty response content or a short placeholder message. The finish reason code is often hidden one field deeper and nobody looks at it if the test script simply counts hits. In this setup, an empty string looks exactly like “the model didn’t find the information.”
The effect is insidious because a zero result fits the intuitive hypothesis. We expect a long document to burden memory, so failure on a long document seems like confirmation. But we’re measuring something completely different: whether the material even passed through the input filter.
This is my opinion, but I believe some published “AI context limit” comparisons are actually measuring moderation. I have no data for this beyond my own measurement, so I treat it solely as a hypothesis, not an established fact.
What Specifically Changed After Model Substitution
After switching to a different model, I didn’t touch anything else. Same document, same six questions, same order, same infrastructure. Hits: 6 out of 6. Times: from 2.3 to 4.9 seconds, depending on how deep in the document the sought information was located.
The 2.3-4.9 second spread is also information. It shows the model actually goes through the material rather than answering from pattern memory. If it were answering with generalities, the times would be more uniform.
Most importantly: since the second model completed the task in full, the task was achievable. The earlier zero wasn’t an assessment of the test’s difficulty.
When to Suspect Moderation Rather Than Context
Several warning signals emerge from my measurement. First: the result doesn’t degrade gradually, it drops sharply. A real memory problem usually gives a soft transition - first one piece of information is lost, then another. An input filter switches binarily.
Second signal: the boundary falls at a round, low value. Approximately 2000 tokens is decidedly too little to talk about exhausting the context window of any contemporary model. If the result collapses this early, context is almost certainly not the culprit.
Third: refusal for safe content. Text about plants is a good control material precisely because there’s nothing in it to block. If it too returns with refusal, the cause lies in volume or configuration, not in topic.
How to Check This Yourself
- Log full raw API responses, not just the content field. You’re interested in finish reason codes - in my case these were
content_filterandnative_finish_reason: refusal. - Prepare one document with six hidden pieces of information distributed from beginning to end of the text. Record exactly where each one sits.
- Run the test on a short version, well below 2000 tokens. This is your reference point - if it doesn’t work here, the problem is in the prompt, not the length.
- Increase document volume in jumps and note at what value the first refusal appears. If the boundary falls sharply rather than gradually, think about filters.
- Substitute thematically neutral material - for me it was text about plants. If it too returns with refusal, content isn’t the cause.
- Repeat on a second and third provider. I checked AWS, the producer’s API, and Google. Behavioral consistency across three platforms rules out local configuration error.
- Swap the model for a different one, changing nothing else. If it hits all six pieces of information, you have proof the test was achievable and the earlier zero measured something else.
- Record response times separately for each question. Spread - for me 2.3-4.9 seconds - confirms the model actually searches the material.
Where These Numbers Come From
All values come from my measurement performed before September 21, 2026. I tested one model through three independent access paths: AWS, the producer’s direct API, and Google. Test material: a document with six hidden pieces of information, scaled by volume, plus a control version about plants without technical content. Refusals with codes content_filter and native_finish_reason: refusal occurred for every document above approximately 2000 tokens across all three providers. The alternative model on an identical set of tasks hit 6 out of 6 hidden pieces of information in 2.3-4.9 seconds. I’m not providing model names or exact moderation thresholds because I have no confirmation that provider-side settings remained unchanged after the measurement date. Fragments marked as opinion are my conclusions, not measurement results.
More on this from QUANTSLANT
benchmarks·Qwen-Image-2.1 Alpha Channel: Native Transparent PNG TestWe measured alpha channel output in Qwen-Image-2.1: 22.7% fully transparent pixels and full 0-255 range. How to prompt for transparency and what to watch for.