What Fits Versus What Gets Used

A company's agreement with a supplier lives in a 200-page pack: a term sheet, the master agreement, schedules, correspondence and amendments. Someone pastes the whole thing into an AI assistant with a million-token window and asks a simple question. If we want to end the agreement early next year, how much notice do we owe?

The answer comes back fluent and specific: "90 days, per clause 4.2." The citation is correct. Clause 4.2 does say 90 days.

But on page 180 there is a signed amendment that reads "In Section 4.2, 90 is replaced by 120." The real answer is 120 days, and nothing in the model's reply tells you it never connected page 180 to page 23.

That exact pack is invented, and the failure is a prediction rather than a test run. But every step of it is a failure that researchers have measured, again and again, in every model they tested. Here is the idea behind it. A context window tells you how much text a model can take in. How well the model uses that text is a different number, and it falls as the text gets longer, less literal, and more crowded with look-alikes and old versions. So as windows grow, deciding what goes in matters more, not less.

What the window measures

A model reads text as tokens, chunks a bit shorter than a word. The context window is the maximum number of tokens it can take in for one response. As of October 2026, a million tokens is the standard window for Anthropic's current models and for Google's Gemini 3.8 Flash. That's roughly 1,600 pages of English prose.

Everything counts against that limit: the instructions, every message, every document, the descriptions of the tools the model can use, and its own output. In a chat, most of it is your document. In an agent, the software around the model (the harness) fills the window with much more. Anthropic has described a setup where the tool descriptions alone took about 55,000 tokens before the conversation started. In an agent request, the passages that actually answer the question can be under 1 percent of what the model receives.

Think of the window as the floor area of a warehouse. It tells you how much stock fits, and nothing about how fast a picker finds the right box among look-alikes. The analogy breaks in one useful place: a slow picker is still accurate, while a model stays just as fast, gives no signal about what it skimmed, and when it misses the evidence it tends to answer anyway.

The test that made long contexts look solved

The test most people have seen is the needle in a haystack. Hide one sentence (the needle) in a long, unrelated text (the haystack), ask a question whose answer is that sentence, and repeat at many lengths and depths. For frontier models the results have been almost perfect for a while. OpenAI reported "very good performance" for GPT-4.1 on this test across its full million-token window.

The same guide carried a warning that gets quoted less: performance can degrade when the model must find many items or reason over the whole context. That warning is the real story, because the needle test flatters in four ways that real work doesn't.

• One fact. Real questions often need two or more facts combined.
• Shared words. The question usually repeats the needle's wording, so the model can match text instead of meaning.
• An unrelated haystack. A sentence about a magic number stands out among essays. A clause in a contract stands among hundreds of clauses that look like it.
• One version. The haystack never contains an older, contradictory copy of the needle.

What happens when the test gets honest

Researchers spent three years removing those shortcuts one at a time. The clearest result comes from a benchmark called NoLiMa, from LMU Munich and Adobe Research, which keeps the needle and takes away the shared words.

A typical needle reads "Actually, Yuki lives next to the Semper Opera House." The question is "Which character has been to Dresden?" To answer, the model has to know that the Semper Opera is in Dresden (which it does; it answers easily when the text is short) and make that link while reading.

Take one model, Llama 3.3 70B, with 32,000 tokens of haystack around the needle. When the question repeats the needle's words, it scores 98.5 percent. When the question needs one associative step, like Dresden to the Semper Opera, it drops to 56 percent. With two steps, 26 percent. Same model, same haystack, same length. Length barely touches the literal version and crushes the version that needs a single inference.

Then the authors added one irrelevant sentence that shares the question's keyword: "There was an article about Dresden in the daily newspaper." With that sentence present, GPT-4o kept its short-text accuracy only up to about 1,000 tokens of input.

The paper defines a model's effective length as the longest input at which it keeps 85 percent of its own short-text score. On these low-overlap questions, every model tested had an effective length between 16 and more than 1,000 times smaller than its advertised window. GPT-4.1, which claims a million tokens, came out at 16,000.

Scroll sideways to see the whole figure

Here's the most useful detail. NVIDIA's RULER benchmark, which keeps literal matching but asks for several facts at once, gave Gemini 1.5 Pro an effective length above 128,000 tokens. NoLiMa gave the same model family 2,000. The tests differ in setup, but setup differences wouldn't plausibly produce a gap of more than sixtyfold. Effective length belongs to a model, a task and a kind of text together. When someone quotes a single number for "how much context a model can really use", that number doesn't exist.

A 2025 study by Chroma tested 18 models, including Claude, GPT and Gemini, on deliberately simple tasks and gave the pattern a name, context rot. Performance fell with input length in every experiment, even when the task was just copying out a sequence of repeated words. In the authors' words, "even a single distractor reduces performance". Anthropic's own documentation now says it plainly: "more context isn't automatically better. As token count grows, accuracy and recall degrade."

Capacity belongs to the window. Use belongs to the window, the text in it and the question together.

Models have improved since these studies; GPT-4.1 roughly doubled GPT-4o's effective length. The curves for current models are probably higher. Nothing found shows them flat.

Why more text makes the right line harder to use

The core mechanism is a competition. At every step of writing, a model decides how much weight to give each earlier token, and those weights are normalized so they add up to one. That's what engineers call attention. Relevant text has to win a share of a fixed budget against everything else in the context.

A question that repeats the needle's exact words produces a strong, unmistakable match, which is why the literal needle test survives a million tokens. An associative link such as Dresden to Semper Opera produces a weaker signal, and a weak signal is easier to drown as the number of competitors grows. That's the explanation the NoLiMa authors offer, and Anthropic's engineers describe the same idea as a limited "attention budget". It fits the data well, though nobody has traced it component by component inside a model.

Four more mechanisms pile on top, and each one shows up in real documents:

Distractors look relevant. Text that shares words or topic with the question draws weight the answer needed. Real collections are full of them: drafts, neighboring clauses, last year's policy.

Old and current versions look the same. A model has no built-in way to know that one paragraph supersedes another. Everything in its context carries the same apparent authority unless the text itself says otherwise. Which version is signed, which is current and which is a draft are facts about the world, and no amount of model capability infers them reliably from text that doesn't state them.

Summaries drop details. Long sessions survive by compaction: the harness replaces older turns with a summary. Anthropic's documentation says documents and fetched pages inside the summarized turns are gone afterward. A later step may need exactly what the summary judged unimportant.

Gaps get filled. When the evidence is buried, a model asked a question tends to produce an answer anyway, assembled from nearby text or general knowledge. For most work, "I can't find this" is the more valuable failure, because someone can act on it.

The contract, traced

Go back to the 200-page pack, about 125,000 tokens. Every one of those mechanisms is in it.

Scroll sideways to see the whole figure

First, the model has to connect "end the agreement early" to clause 4.2's "terminate for convenience". The phrases share no content words, exactly the situation NoLiMa measured. Second, it has to find that the amendment on page 180 changes clause 4.2. The amendment never mentions termination or notice; it refers to the clause only by number, a second hop. Third, it has to resolve a conflict: 90 appears twice (in the clause and in the term sheet), 60 appears once in an unsigned draft, and 120 appears once, in the signed, later document. Fourth, it has to ignore that draft, which talks about termination and so shares more words with the question than the real amendment does.

Every fix changes what reaches the model. An exact search for "4.2" across the pack returns the clause and the amendment, while a search by meaning would probably rank the draft above the amendment. Marking each document's status before loading (signed or draft, date, what it amends) removes the conflict the model would otherwise guess at. Asking for quotes with page numbers before the answer makes a missed amendment visible. A well-chosen context of perhaps 4,000 tokens, holding the clause, the amendment and a status table, turns a hard long-context problem into an easy short one.

Most long-context failures are fixed upstream, in what the model receives.

The case for loading everything, and where it stops

Capable people argue the opposite, with good reasons. Windows went from about 4,000 tokens in 2023 to a million as a default three years later. Each generation uses long inputs better. Retrieval, the machinery that searches a collection and loads only the passages likely to matter, has to be built and maintained, and its misses are silent. And prompt caching, where the provider keeps an already-read prefix ready for repeat questions, has removed most of the cost penalty. At October 2026 list prices for Claude Opus 5.5, one question against the whole contract pack costs about 50 cents. Later questions against the cached pack cost about 2.5 cents, roughly the same as a carefully selected 6,000-token context.

I think part of that argument is right. For small, coherent material, and for tasks that genuinely need all of it (summarizing a report, checking a document against itself), loading everything is often the best choice today, and that range will widen. Elaborate document-chunking tuned to one model's weaknesses will age badly.

But three things don't follow. The measured difficulty doesn't come from size alone; it comes from low overlap, multiple steps and look-alike distractors, and adding more of your own documents adds exactly those. Improvement so far has raised the curves without flattening them: GPT-4.1 still lost about a third of its accuracy between short inputs and 128,000 tokens on NoLiMa. And no reader, however good, can settle a contradiction between versions when nothing in the text says which one governs. A better reader only guesses more fluently.

The second-order effect matters for anyone running these systems. As caching makes long contexts cheap, cost stops being the argument for selecting what goes in, and accuracy becomes the argument. A team that built retrieval to save money may reasonably drop it when caching arrives. If nobody measures accuracy, the loss shows up only as answers that are a little more often wrong.

This is where a distinction from Same Model, Different Results earns its keep. Some of the work around a model is scaffolding: it compensates for a current weakness and should shrink as models improve. Chunking documents so a weak reader copes is scaffolding. Some of it is a wall: it encodes facts about the world and stays. Recording which version is signed, which document supersedes which, and who may see what is a wall. Better models will need less of the first and exactly as much of the second (Write for the Model That Doesn't Exist Yet).

What to do

Test it on your own documents before you trust a window size. Build a small evaluation set: 20 real questions whose answers you know and where they live. Run each with only the relevant document, with about 50,000 tokens of surrounding material, and with the full collection. Phrase some questions in the document's words and some in the words a colleague would use. Include a stale version. Score three outcomes separately: right with the right citation, an explicit "not found", and wrong. Averaging a confident wrong answer with "not found" hides the outcome that matters most. At Opus 5.5 list prices the core test costs about $43. If the full collection matches the selected context on your questions, you've earned the right to load everything for that task (Build an Evaluation From Your Own Work).

If you paste long documents into an assistant, paste the part that matters when you know it, ask narrower questions, and ask for quotes with page numbers. Then ask what the documents leave unanswered or contradict; that question surfaces gaps a direct question papers over.

If you build, treat the context as a budget with an owner. Put the question at the end of long documents, prefer exact search for identifiers and section numbers, and tell the model that "not found" is an acceptable answer and that conflicts should be reported, not silently resolved. Log what went into every request. The first question after a bad answer is whether the evidence was there, buried, or contradicted.

If you were sold on a million-token window, read it like the floor area of a warehouse: real, useful and silent about whether anyone can find anything. Ask vendors for results on tasks like yours, at the lengths you'll use. Then budget for the work the window doesn't do. In the contract, every fix depended on knowing which documents were signed, which superseded which, and where each answer lives. That preparation is rarely in the project plan, and in organizations with years of accumulated versions it decides how reliable the system is.

The window will keep growing. The question it can't answer, what in all that text is current, relevant and true, stays with whoever prepares the input.

Check yourself

Why does the needle in a haystack test make long contexts look solved?

It uses one fact, shared words, an unrelated haystack and one version. Real work has several facts, different wording, look-alike text and old versions.

Why did the model fall from 98.5 to 56 percent when the question needed one step?

A literal match is a strong signal. An associative link like Dresden to the Semper Opera is weaker and is easier to drown as more competing text surrounds it.

Why can't a better reader settle 90 days versus 120 days in the contract pack?

Nothing in the text says which version governs. A better reader only guesses more fluently, so the fix is to record upstream which document is signed and current.