A context window is a capacity claim, not a recall promise
A million-token window means a million tokens will be accepted without an error. It does not mean the model will find the sentence you buried at token 400,000 — and the published number cannot tell you which.
A context window is the maximum number of tokens a model will accept in one request. It is an input limit enforced by the serving stack, not a statement about comprehension. Measured retrieval accuracy inside that window is uneven: models reliably recall material at the beginning and end of a long input and degrade in the middle, an effect documented across model families and still present as windows grew from thousands of tokens to millions.{{1}} Benchmarks that vary task difficulty rather than only input length find that effective context — the length at which a model still performs the task — is frequently a fraction of the advertised figure.{{2}} Read the declared number as the ceiling on what you may send, then measure recall on your own documents at the length you actually use. Two models with the same published window can behave very differently at 80% of it.
What the number literally means
The context window is a hard limit in the serving path. Send one token more and you get an error, not a worse answer. That is the whole of what the number guarantees: an acceptance threshold, enforced before inference begins.
It is easy to read it as something richer — a working memory, an attention span, a promise that everything you send is available to the answer. None of that is claimed by the figure, and none of it is tested by the figure. Vendors publish the limit their infrastructure enforces. Whether the model uses the middle of that window is a separate, measurable, and frequently unflattering question.
This matters most exactly where long context is most attractive: dropping in a whole contract, a full codebase, a year of meeting notes, and expecting the model to behave as if it had read all of it. It accepted all of it. That is not the same claim.
- The window is shared. System prompt, tools, retrieved documents, conversation history, and the reply all draw on the same budget.
- Output has its own ceiling. Maximum output tokens is a separate limit and is usually far smaller than the input window.
- Reasoning consumes it too. On reasoning models, internal tokens are billed and counted even when they are never shown to you.
- Tokens are not words. Code, non-English text, and structured data tokenize less efficiently than English prose, sometimes by a factor of two or more.
Where recall degrades, and why the shape matters
The most replicated finding in long-context evaluation is positional. When a single relevant fact is placed at varying depths in a long input, accuracy is highest when it sits near the start or the end and lowest when it sits in the middle — a U-shaped curve reported across model families and prompt formats.[1] The practical translation is blunt: material in the middle of a very long prompt is the material most likely to be missed.
Single-fact retrieval is also the easiest possible version of the task. The widely used needle-in-a-haystack probe hides one distinctive sentence in filler and asks the model to find it.[3] Passing it says a model can locate one salient string. It does not say the model can hold several facts at once, notice that two of them conflict, or track something across a hundred pages.
Evaluations that vary difficulty rather than only length find the gap this hides. RULER extends the haystack idea with multi-hop tracing, multiple needles, aggregation and question answering across controlled lengths, and reports that models advertising very large windows often fall below their short-context baseline well before reaching the advertised limit.[2] The number on the pricing page and the length at which the model still does your task are different numbers, and only one of them is published.
| What you are asking for | How it behaves as input grows | What to do about it |
|---|---|---|
| Find one distinctive fact | Holds up longest; still weakest in the middle | Safe to rely on, but place critical text at the start or end |
| Find several facts and combine them | Degrades noticeably earlier than single-fact recall | Retrieve first, then send a shorter prompt containing only what matters |
| Notice that two passages disagree | Degrades early; absence of a flag is not absence of conflict | Ask for the conflict explicitly, or compare passages in separate requests |
| Summarize faithfully across the whole input | Coverage skews toward the ends of the document | Chunk, summarize each chunk, then summarize the summaries |
| Count or aggregate across many items | Unreliable at any substantial length | Do the arithmetic in code, not in the prompt |
How to read a published context figure
Treat the declared window as one field in a specification rather than a summary of the model. Three questions get you most of the way to an honest reading.
First, what is the output limit? A model that accepts a million input tokens and returns at most sixty-four thousand is a reading instrument, not a writing one. Second, what does length do to price? Several providers apply a higher rate above an input threshold, and that rate frequently applies to the entire request rather than only the tokens above the line — so crossing the threshold can reprice work you thought was cheap. Third, is the number even attested? Catalogs carry unknowns, and a profile that reports a window without a source is reporting a rumour.
Then run the only test that settles it: take a document of the length you actually work with, hide three facts in it at the start, middle and end, and ask questions that require all three. Do it on each candidate model. The exercise takes twenty minutes and reliably reorders shortlists that were built from specification sheets.
- Ask for the effective length, not the maximum — the length at which your task still passes.
- Test with your own documents. Public haystack results use filler text that is nothing like a contract or a codebase.
- Put the instruction and the most important material at the end of the prompt, where recall is strongest.
- Measure again after a model version changes. Effective context is not a stable property of a model name.
When a huge window is the right tool anyway
None of this makes long context a gimmick. It makes it a tool with a shape. Large windows are genuinely the right answer when the material is cohesive and has to be reasoned about as a whole — a single long legal document, one large source file being refactored, a transcript where the meaning depends on the sequence. Chunking those destroys the thing you needed.
They are the wrong answer when the material is a heap of loosely related documents and you need three paragraphs from it. There, retrieval is not a workaround for a small window; it is the better method at any window size. It costs less, runs faster, and — because it puts a short, relevant prompt in front of the model — usually produces a more accurate answer than dropping the whole corpus in and hoping.
The failure worth naming is using a large window as retrieval: sending everything because the limit allows it, and treating acceptance as evidence of comprehension. That is the habit the positional findings should break.
Frequently asked questions
What is a context window in simple terms?
It is the maximum amount of text, measured in tokens, that a model will accept in a single request. Everything counts toward it: the system prompt, your files, the conversation so far, tool definitions, and the reply. Exceed it and the request is rejected outright rather than answered less well.
Does a bigger context window mean better answers?
Not by itself. A larger window means more text is accepted. Measured retrieval accuracy inside long inputs is uneven, with the middle of the input the weakest position, and benchmarks that vary task difficulty find models often falling below their short-context performance well before the advertised limit.
How many words is a million tokens?
Roughly 700,000 to 750,000 words of English prose, though the ratio is only a rule of thumb. Code, non-English languages, and structured formats such as JSON tokenize less efficiently, sometimes needing two or more tokens per word.
Should I use a long context window or retrieval?
Use long context when the material is one cohesive thing that has to be reasoned about as a whole. Use retrieval when you need a few relevant passages from a large, loosely related corpus — it is cheaper, faster, and usually more accurate, because a short relevant prompt outperforms a long mostly-irrelevant one.
Why does the same model behave differently at the same length?
Position, task type, and prompt structure all matter. The same fact is recalled more reliably at the start or end of an input than in the middle, and a task requiring several facts to be combined degrades earlier than one requiring a single lookup.
Sources and evidence
Primary and authoritative sources used for factual claims. Company research and executive forecasts are labeled as such in the article.
- 1Lost in the Middle: How Language Models Use Long ContextsLiu et al., Stanford / UC Berkeley / Samaya AI · 2023-07
- 2RULER: What's the Real Context Size of Your Long-Context Language Models?Hsieh et al., NVIDIA · 2024-04
- 3Needle In A Haystack — pressure testing LLMsGreg Kamradt · 2023-11
- 4