A larger context window changes how much material a language model can accept. It does not decide which material deserves attention, which copy is current or which documents a user may see. Those remain system decisions.
The choice is therefore not “long context or retrieval”. A full-context design supplies the eligible corpus, or a large part of it, and leaves the model to locate and combine evidence. A retrieved-context design selects a smaller set first. A hybrid uses retrieval to choose evidence and a long window to preserve enough of it for the task. The right boundary depends on the shape of the question, not on capacity alone.
Capacity is not answer quality
Full context has a clear advantage when evidence is distributed and relationships matter. A contract review may need a definition, an exception and an appendix together. Selecting isolated passages can break exactly those dependencies. When the corpus is small, stable and authorised, sending it whole also removes a retriever that could omit the decisive passage.
That advantage can weaken when most input is irrelevant. The model must still discriminate between evidence and distractors. In Lost in the Middle, Liu and colleagues changed the position of the answering passage without changing the desired answer. Several tested models performed worse when that passage sat in the middle, although the effect varied by model and task; some models handled the synthetic key–value test nearly perfectly. This is evidence for position-sensitive behaviour in those evaluations, not a universal law of long-context models.
A separate retriever–reader experiment in the same paper found that answer accuracy levelled off before retriever recall. Adding documents beyond 20 produced little improvement for the tested readers. More recall can help only if the reader can use what arrives.
Retrieval and long context can also work together. Xu and colleagues compared retrieval-augmented and extended-context models across question answering, query-based summarisation and few-shot tasks. Retrieval helped their tested long-context models, and their best configuration combined a 32K window with retrieved chunks. Yet increasing from five or ten chunks to 20 did not always help, even when the chunks fitted. These results support measuring the combination rather than treating either technique as a substitute for evidence design.
Ranking determines the evidence budget
A retriever makes its selection through a sequence of compromises. Chunking sets the unit of meaning. Short chunks improve matching precision but can detach a claim from its conditions; long chunks preserve context but dilute the match and consume more tokens. Keep source identifiers, headings and neighbouring text so a selected passage can be interpreted and inspected.
Candidate generation then balances lexical and semantic matching. BM25 can reward exact names and rare identifiers. A dense dual encoder can match a paraphrase by comparing query and passage vectors. Dense Passage Retrieval showed that these approaches behave differently across datasets and that combining their candidate sets sometimes improved retrieval. Neither score establishes that a passage supports an answer.
A reranker can examine a smaller candidate set more closely, but it adds model work and another failure point. Set the final k by downstream answer quality as well as recall at k. Multi-hop questions may need several independently relevant passages; a narrow factual question may need one. The useful stopping point is where another passage no longer improves answers enough to justify its cost and noise.
Full context does not escape selection. It leaves that selection inside the model, where omissions are harder to measure. Retrieval makes part of the decision observable: teams can inspect candidates, scores and excluded passages before debugging the generator.
Deduplicate for diversity, then order deliberately
Similarity search often returns near copies: overlapping chunks, repeated boilerplate or several versions of one policy. Exact deduplication saves tokens. Near-duplicate detection can make room for distinct evidence. Neither should collapse documents that differ in date, scope or authority. A changed clause is not redundant merely because the surrounding text is unchanged.
Diversity also has a failure mode. Removing all overlap may discard corroboration or separate passages that form one reasoning chain. Group duplicates by source and version, retain their provenance and select across evidence needs rather than maximising novelty for its own sake.
Context order is another model input. Ranking by descending relevance puts the strongest candidate first, but a chronological reconstruction or document structure may be easier for the model to use. Place essential instructions and the query where the chosen model handles them reliably. Query-aware contextualisation improved the synthetic retrieval task in Liu’s study but did little for multi-document question answering, so copying one prompt layout across tasks is unsafe.
Evaluate order directly. Keep the evidence set fixed, move each decisive passage through the prompt and record best, worst and median answer quality. Include contradictory and duplicated evidence. A system whose answer depends on an arbitrary shuffle has a robustness problem even when its average benchmark score looks sound.
Measure the whole request, not the window
Full context usually increases input processing and billed prompt tokens. Conventional dense self-attention has quadratic computation and memory growth in sequence length, but that statement does not describe every efficient-attention implementation or end-to-end serving stack. Kernel design, caching, batching and model architecture all affect production latency.
Retrieval adds query encoding, index search, possible reranking and prompt assembly. It also shifts work offline into extraction, embedding and index maintenance. Those costs can be shared across requests. A shorter prompt does not guarantee a faster answer if the retrieval chain is slow; a long prompt does not guarantee a slower total request if it replaces several retrieval or generation rounds. Google’s Gemini 1.5 announcement made this distinction practical: its million-token configuration was a limited private preview, and Google warned early testers to expect longer latency while optimisation continued.
Compare configurations on the same tasks with the same generator and output policy. Measure time to first token and total response time, retrieval and reranking time, input tokens, index memory and cost per answered request. Report distributions rather than a single favourable prompt. Include cache hits and misses if production traffic uses caching.
Freshness needs an ingestion policy
An external index can be updated without retraining the generator. Lewis and colleagues demonstrated this by swapping Wikipedia snapshots: matched snapshots answered time-sensitive leader questions much more often than mismatched ones. That is evidence for replaceable knowledge, not automatic freshness.
A current answer requires source synchronisation, deletion handling and a known index version. Record source and update timestamps, then test questions whose answers change. Full context faces the same constraint if someone assembles a stale snapshot by hand. Retrieval makes updates operationally easier only when the ingestion pipeline actually delivers them.
Enforce access control before generation
Permissions are not a relevance feature. Apply the user’s identity and group memberships as access-control filters when searching or assembling the corpus, before any text reaches the model. Carry the same rules through reranking and neighbouring-passage expansion. Otherwise a permitted chunk can pull in a restricted paragraph.
Do not retrieve forbidden material and remove it after ranking. That can distort counts, leak information through logs or intermediate results and leave too little evidence for an answer. Filter first, then rank the authorised candidates. Amazon Kendra’s 2020 launch already described connectors that maintained document access rights and search results limited to documents the requester could access.
Revocation and deletion belong in the ingestion tests as well as the security design. A denied-document suite should verify absence from the prompt, answer, citations and logs. The generator is not an authorisation boundary.
Choose by evidence shape and operating constraints
Use full context when a modest, stable corpus must be read as a whole and its cost is acceptable. Use retrieval when the corpus is large, changing or permissioned, and questions usually have localisable evidence. Use a hybrid when selected passages still need a wide window for cross-document reasoning.
Evaluate all three against representative questions. Track evidence recall separately from answer support, and freshness separately from relevance. Permute evidence positions, revoke permissions and introduce stale or conflicting versions. The result is an architecture decision grounded in observed failures. A larger window raises the ceiling on what can be supplied; selection still determines what should be.
Sources and further reading
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — external memory, replaceable indices and time-sensitive knowledge, v4, 12 April 2021.
- Karpukhin et al., Dense Passage Retrieval for Open-Domain Question Answering — dual-encoder ranking, lexical retrieval and candidate reranking, v3, 30 September 2020.
- Liu et al., Lost in the Middle: How Language Models Use Long Contexts — position-controlled evaluations and the limits of adding documents, v3, 20 November 2023.
- Xu et al., Retrieval Meets Long Context Large Language Models — controlled comparisons of retrieval, extended windows and their combination, v2, 23 January 2024.
- AWS, AWS Announces General Availability of Amazon Kendra — source synchronisation and permission-aware enterprise search, 11 May 2020.
- Google, Introducing Gemini 1.5 — context-window availability and latency considerations in the February 2024 announcement, 15 February 2024.