A language model stores some knowledge in its parameters, but those parameters do not preserve a reliable source for each fact or offer a direct way to revise one. Retrieval-augmented generation (RAG) adds an external corpus at inference time. A retriever selects passages from that corpus, and a generator produces an answer conditioned on them.
This changes the evidence available to the model, not the nature of generation. The answer remains a probabilistic synthesis of the query, retrieved text and knowledge already encoded in the model. RAG can therefore reduce unsupported answers without eliminating them. Its practical value lies in making knowledge replaceable and inspectable; its practical risk lies in presenting that inspection trail as proof of truth.
The index determines what the model can know
Grounding starts with ingestion, long before a user asks a question. A system must extract usable text, divide it into retrieval units and attach enough metadata to recover the original context. The original RAG implementation split a Wikipedia dump into 21 million disjoint passages of 100 words. That fixed size made indexing tractable, but it was a design choice rather than a natural unit of evidence.
Chunking sets a compromise. Small passages give precise matches and fit more distinct sources into a prompt, yet can sever a definition from its qualifying conditions. Large passages preserve context but dilute the embedding and consume more of the generator’s context window. Boundary-aware extraction matters as much as length: a heading, table row or footnote may carry the meaning that a plain text fragment lacks.
A dense retriever then maps every passage to a vector. In Dense Passage Retrieval, separate encoders produce a passage vector offline and a query vector at run time; their inner product supplies the similarity score. Because passage vectors are precomputed, nearest-neighbour search can select candidates from millions of passages without running the passage encoder for every query. Approximate search usually trades some recall for lower query latency; its memory use depends on the index.
The vectors alone are not knowledge in a directly usable form. Every entry must retain a stable source identifier, text, corpus version and access rules. If the document encoder changes, old vectors are not necessarily compatible with new queries. Re-embedding and rebuilding become migration work, not a transparent model upgrade.
Top-k is an evidence budget
At query time, the retriever ranks passages and returns the top k. This small set is the generator’s explicit evidence budget. If the relevant passage is absent, adding instructions to remain factual cannot restore it. The generator may still answer correctly from parametric memory, but that answer will not be grounded in the retrieved corpus.
Increasing k can improve coverage, especially when a question needs several pieces of evidence. It also adds search work, prompt tokens and generation cost. More passages introduce more opportunities for distraction or contradiction, so the best k depends on the corpus, query and architecture. The original RAG experiments found different behaviour as k increased for sequence-level and token-level formulations; there is no universal setting to copy.
Dense similarity captures paraphrases that keyword matching can miss, but it does not make every close vector relevant. Rare names, identifiers and exact terminology can favour lexical methods such as BM25. Dense Passage Retrieval found benefits from combining the two approaches in some settings. Retrieval evaluation should ask whether the returned set contains evidence that answers the query, not merely whether the first result looks topically related.
Generation does not obey the evidence
The original RAG model paired a Dense Passage Retriever with BART. It concatenated the query and retrieved text, then generated tokens conditioned on both. RAG-Sequence treated one latent document as responsible for a whole output; RAG-Token allowed different retrieved documents to influence different tokens. Later systems used other arrangements. RETRO, for example, retrieved neighbours for chunks of an autoregressive sequence and incorporated them through chunked cross-attention. These are different architectures pursuing the same broad separation between parametric and non-parametric memory.
That separation does not establish precedence. The generator can quote a passage, paraphrase it, combine it with other passages or rely on something learned during pretraining. RAG’s authors reported more factual generation than a parametric baseline, but described a reduction in hallucination rather than its removal. They also produced some correct answers when no retrieved document contained the answer. Parametric memory can fill a gap; it can also bypass the evidence trail.
Synthesis creates further failure modes. A paraphrase may drop a date qualifier, attach a number to the wrong entity or turn a possibility into a certainty. Several individually supported statements do not guarantee a supported conclusion. Retrieval-conditioned generation needs answer-level testing even when retrieval recall is high.
Freshness is a corpus operation
An external index can be replaced without retraining the generator. RAG demonstrated this by swapping Wikipedia snapshots and asking about leaders who had changed between them. Matched snapshots usually produced the leader for the corresponding period; mismatched snapshots did not. This is concrete evidence for updateable knowledge and for the damage caused by a stale index.
Replacement is not a freshness guarantee. A production corpus needs ingestion schedules, deletion handling and a way to reconcile sources that describe different points in time. Document timestamps can help a reader interpret evidence, but they do not establish which source is authoritative. Retaining an obsolete copy may be necessary for historical questions while harmful for questions about the present.
Model changes create a second form of staleness. REALM periodically re-embedded and re-indexed its corpus during pretraining because document-encoder updates made the existing search index inconsistent. The original RAG work avoided that training cost by freezing its document encoder. A deployed system faces the same compatibility issue whenever it upgrades embeddings: content freshness and vector compatibility require separate version checks.
Name the failure before choosing the fix
Four outcomes are often grouped under hallucination, although they have different causes and remedies:
- Retrieval failure: the corpus contains useful evidence, but top-k search misses it. Investigate extraction, chunk boundaries, query representation and recall before changing the generator.
- Stale evidence: retrieval returns the intended passage, but its information is outdated. Repair the ingestion and index-refresh process; better prompting will not update the text.
- Unsupported generation: the answer extends, misreads or contradicts the supplied passages. Test generation with known-good evidence, adjust the answer policy and allow abstention when support is insufficient.
- Misleading attribution: the answer displays a real source, but that source does not support the claim attached to it. Check the claim–citation mapping rather than counting links or retrieved documents.
A single answer can exhibit all four. Diagnosing only the final text hides where the evidence chain broke.
Citation is an inspection aid, not a verdict
Retrieved text offers provenance because people can read it. That is weaker than a faithful explanation of why the generator produced an answer, and weaker still than proof that the answer is true. A bibliography appended to a paragraph establishes neither relationship.
WebGPT made this distinction operational by collecting references while browsing and asking evaluators to annotate the support for individual claims. Its authors also warned that citations could make answers appear more authoritative and that the system could select convincing references without assessing the evidence fairly. The interface helps inspection; it does not settle the inspection.
GopherCite pursued a stricter mechanism. Constrained generation made its quotations verbatim extracts from retrieved documents, while human raters judged whether each quotation supported the associated claim. Verbatim matching prevents a fabricated quotation. It does not establish entailment, source reliability or freedom from cherry-picking. GopherCite’s own TruthfulQA analysis separated claims supported by evidence from claims true of the real world.
Claim-level verification should therefore preserve a precise mapping from each material statement to its evidence. Check that the passage exists, addresses the same entities and period, and supports the statement’s qualifiers and strength. Review every consequential claim, not just the sentence carrying a link. Some conclusions need several passages; negative and statistical claims may not be verifiable from a concise quotation at all. In those cases, an honest system should show the gap or decline to answer.
Production needs two evaluation loops
Evaluate retrieval independently with representative queries and labelled evidence. Measure whether adequate passages appear within k, including queries with no answer in the corpus and facts that change over time. Record the corpus snapshot, embedding model, index version and access-control result so a run can be reproduced.
Evaluate answers separately by splitting them into claims and checking support against the exact context supplied to the generator. Include conflicting passages, missing evidence and sources of uneven quality. Human review is appropriate for these judgements because textual overlap cannot determine whether evidence warrants a conclusion. Track abstention alongside answer quality so the system cannot improve its apparent reliability by answering only easy cases.
RAG moves part of a language model’s knowledge into infrastructure that teams can update and inspect. It does not make the resulting answer self-verifying. Reliable use depends on preserving the whole chain from source to passage, from passage to claim, and from claim back to evidence that a reader can test.
Sources and further reading
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — the RAG architecture, index hot-swapping and generation analysis, v4, 12 April 2021.
- Karpukhin et al., Dense Passage Retrieval for Open-Domain Question Answering — dual-encoder retrieval, passage indexing and lexical comparisons, v3, 30 September 2020.
- Guu et al., REALM: Retrieval-Augmented Language Model Pre-Training — asynchronous index refreshes and adaptation to a new corpus, 10 February 2020.
- Borgeaud et al., Improving Language Models by Retrieving from Trillions of Tokens — RETRO’s chunked retrieval and cross-attention architecture, v2, 11 January 2022.
- Nakano et al., WebGPT: Browser-Assisted Question-Answering with Human Feedback — references, claim annotation and citation-related overreliance, v3, 1 June 2022.
- Menick et al., Teaching Language Models to Support Answers with Verified Quotes — verbatim evidence and the distinction between supported and true claims, 21 March 2022.