How much AI models actually remember

A million-token context sounds like a promise that the model will read an entire book and remember everything. Reality is different: a large window tells you how much you canfit in, not how much it will find will find. I test this with my own benchmark: I hide a unique code in a long document and ask the model to find it. The result is binary-either it returns the code or it doesn't.

last run: 2026-11-09 05:48:19 · two hiding positions (25% and 75% of document) · exact match

Model	8 k tokens	32 k tokens	128 k tokens
Claude Opus 5	no responsetime limit exceeded	no responsetime limit exceeded	no responsetime limit exceeded
GPT-5.6 Terra	no responsetime limit exceeded	no responsetime limit exceeded	no responsetime limit exceeded
Kimi K3	no responsetime limit exceeded	no responsetime limit exceeded	no responsetime limit exceeded
DeepSeek V4 Pro	no responsetime limit exceeded	no responsetime limit exceeded	no responsetime limit exceeded

How the test works

I build a document from numbered paragraphs with neutral content (archival protocols), and insert one sentence with a code in a chosen location-a code that cannot be guessed or learned, because it's generated from a checksum of the date and test parameters:

Special paragraph. The backup archive access code is NB-A3F91C and is valid only in this document.

Then I ask the model to provide just the code. I hide it twice: once at one-quarter through the document, once at three-quarters-because models typically remember the beginning and end best, but lose the middle. "Found" means two hits out of two, "partial" means one, "lost" means zero. I also measure response time, because with long context it can grow from seconds to minutes, which matters if you're building a product on top of it.

What this test does not measure

Finding a fact is the simplest task in a long context. Harder is connecting several pieces of information scattered across a document and reasoning from the whole - a model that finds code can still lose the thread when a question requires combining three paragraphs from different places. Treat this result as a floor, not a ceiling: if a model fails this test, it will certainly fail harder ones. In your own deployments, always test on your documents, because text structure matters.

I repeat the test weekly and archive the results - this way we'll see if models start losing more after silent updates than before. You'll find prices for models in this table at cost calculator, and their full profiles in the hubs: Opus 5, GPT-5.6, Kimi K3. When citing results, include "quantslant.com, context window test" and the run date.