AI Context Window Test: Two Models Failed at 25,000 Tokens

👁 119 reads

Every AI provider today boasts context windows of a million tokens. It sounds like a promise that you can feed a model an entire book and ask about details. We decided to test whether this actually works, and we started cautiously: not with a million tokens, but with twenty-five thousand tokens - roughly three percent of the promised window.

Even at this level, two out of five models failed. One didn’t find the hidden sentence, another failed to respond at all twice, despite five minutes of waiting.

How We Tested Context Window Performance

We built a long text from our own published articles - natural language, not generated filler. Ninety thousand characters of continuous text.

In the middle, we hid one sentence that had no business being in that text: information about a service code for an attic mailbox. We inserted the sentence at three depths: after ten percent of the text, at the halfway point, and after ninety percent. Then we asked each model to provide just the code.

This tests one thing: whether the model actually read the entire text, or just the beginning and end.

AI Model Context Window Test Results

The entire test cost 1.31 dollars.

Three of Five Passed, But Costs Varied Dramatically

Gemini 3.8 Flash, Kimi K3, and GPT-6 Astra found the hidden sentence at every depth. The difference is in the bill.

GPT-6 Astra charged 1.00 dollar for three queries. Gemini 3.8 Flash did exactly the same thing for 0.059, which is seventeen times cheaper. For a single query, that’s a thirty-three cent difference. For a tool that searches documents daily, it’s the difference between a bill you don’t notice and one you have to justify.

Astra has one advantage that can be decisive in live work: it responded in two seconds, fastest of all five, regardless of where we hid the sentence.

GLM 5.3 Flash Failed to Respond Twice Out of Three

The model that won our non-English grammar test performed worst here. With the sentence hidden at ten percent and at ninety percent, it exceeded the five-minute limit and returned nothing. It only found the sentence inserted exactly at the halfway point.

This is the third time this same model has failed us in the same way. Previously, it burned through its entire token limit on reasoning and delivered zero words when asked for an article, and in a nine-task test it returned empty responses three times and needed 353 seconds.

The cause is the same every time: you can’t disable reasoning in this model. With a short task, this only raises the bill. With longer input, it can completely block the response.

The conclusion is uncomfortable, but necessary: a model can be very good at a language and simultaneously unsuitable for anything that works with longer text.

Most Interesting Discovery: Same Text, Different Token Counts

We sent all models identical text. Each counted it differently, because each divides text into tokens in its own way.

This is a hidden cost that nobody mentions when comparing price lists. Kimi K3 needs forty-three percent more tokens for the same text than Gemini. At identical rates per million tokens, this means a forty-three percent higher bill for exactly the same work.

We ran this test in Polish, a heavily inflected language. Polish is at a disadvantage compared to English here, because models are trained primarily on English texts, and inflectional endings break into more pieces. The worse the tokenization, the more you pay for the same sentence - a cost penalty that applies to many non-English languages.

A Surprise We Can’t Explain

DeepSeek V4 Flash didn’t find the sentence hidden at ten percent of the text - near the beginning - but found it without problems at the middle and end. This is the opposite of what intuition suggests, because the beginning is usually easiest.

We have no explanation for this and won’t invent one. One attempt per depth is too few to talk about a pattern. We note it as a fact from this measurement, not as a rule.

What This Test Doesn’t Tell You

We didn’t test a million tokens, only twenty-five thousand. So we don’t know how these models behave at full context window, and nobody knows until they pay to check.

We also didn’t test comprehension, only retrieval. Finding one sentence is the simplest possible task on long text. Harder questions requiring connecting information from multiple places would perform worse across all models.

And something to remember with every such comparison: the price on a model’s card may not be the price you pay, because the same model is hosted by dozens of companies with their own rates.

Google · Your sourcesQuantslant higher in your Google - one clickAdd to preferred sources →

Practical Takeaways for Long Context AI

For document search, use Gemini 3.8 Flash. Perfect hit rate, three seconds, and a bill seventeen times lower than the most expensive model with the same result. Plus the best tokenization, which lowers cost even before asking a question.

Don’t assume the declared context window works. Test it on your own material by hiding something in it that shouldn’t be there. It takes fifteen minutes and settles more than all the manufacturers’ tables.

Count tokens, not characters. When choosing a model for work with non-English text, the difference in tokenization can cost more than the difference in the price list.

How much these models actually cost for writing rather than reading, we measured separately in our article about how much an AI-written article costs.

Kinetyra