An agent does not reason over its whole history by default. It reasons over the sequence of tokens assembled for the next model call. Instructions, recent messages, retrieved passages, tool descriptions and tool outputs all compete for that space. Everything else may exist in a database or execution environment, but it cannot inform that model call until the system brings it into view.
Context engineering is the control of that movement. LangChain organises the practice into writing, selecting, compressing and isolating context. The useful engineering distinction is sharper: durable state is the record; the prompt is a temporary, task-specific view of that record. A larger window raises the view’s capacity without deciding what belongs in it.
Build a view for the next decision
The context lifecycle starts before a model call. The runtime holds conversation history, the current plan, retrieved knowledge, tool results and pending actions. An assembler chooses a subset, renders it in a deliberate order and reserves room for the response and further tool feedback. After the call, new messages and artefacts update the durable state. The next prompt can therefore differ without changing the underlying record.
MemGPT made this boundary explicit through main context and external context. Its queue manager evicted old messages into recall storage, maintained a recursive summary and let the model page selected records back through function calls. The analogy with virtual memory is useful, but paging is not transparent here. An operating system moves identical bytes; an agent may retrieve the wrong record or replace exact text with a summary.
This makes token counting an input to control flow rather than a reporting detail. Set thresholds for retrieval expansion, compaction and refusal before the hard limit. A policy that waits for overflow has already lost the choice of what to remove. Keep the budget visible to deterministic orchestration code; do not rely on the model to notice pressure after its instructions or evidence have disappeared.
Select for coverage, not similarity alone
Selection begins with the next decision. A request to revise a travel booking needs the active reservation, the new constraint and the applicable fare rule. Similar messages about an older trip may be semantically close but operationally misleading. Retrieval should combine relevance with recency, authority, permissions and task stage. These are separate predicates; an embedding score cannot stand in for all of them.
Candidate generation should preserve the units needed for interpretation. A short passage may match well yet lose an exception defined two paragraphs earlier. Expand it with neighbouring text or linked definitions when the task requires them, and retain source and version identifiers. Deduplicate repeated tool observations, but do not collapse conflicting versions as though repetition were the only difference.
Selection can also change during a trajectory. Anthropic’s research system used multi-step search that reacted to earlier findings rather than fetching once against the original query. The same principle applies to memory and tool descriptions: inspect the result, identify the unresolved question, then retrieve again. Dynamic retrieval spends extra calls to reduce the chance that one initial ranking determines the whole run.
More candidates still do not guarantee coverage. A retriever can omit the only decisive passage, and a model can overlook evidence that arrived. Track retrieval recall where labels permit, then inspect the final rendered context separately. That distinction tells you whether to fix the index, the assembly policy or the model’s use of evidence.
Treat compaction as an editorial decision
Compaction removes information. Trimming drops whole messages; summarisation rewrites them; token-level pruning keeps a subsequence. Each method can reduce cost and latency while removing the qualifier, identifier or failed attempt that matters later. A compact history is a working hypothesis about future relevance, not a lossless memory.
Make that hypothesis explicit. A trajectory summary should preserve the objective, unresolved questions, binding constraints, decisions and their sources. It should distinguish completed actions from proposals and retain exact identifiers where a later tool call depends on them. Store the summary’s coverage range and links to the raw records so a subsequent step can expand it. Otherwise the system cannot distinguish a fact that never occurred from one that compaction discarded.
LongLLMLingua offers a more selective mechanism than uniform truncation. It ranks documents relative to the question, allocates different compression budgets and uses a smaller language model to guide token retention. It also reorders retained documents and recovers some original subsequences in generated answers. The important idea is question-aware retention, not a universal compression ratio. Its authors note that changing the question requires recompression and adds computation. A reusable cache and a query-specific compressed view serve different purposes.
Summaries introduce a further risk: fluent prose can hide uncertainty. Require the compactor to mark unresolved conflicts rather than reconcile them silently. Where an action is costly or irreversible, reconstruct the relevant evidence from primary records instead of acting on the summary alone.
Isolate bulk without losing the trail
Tool output often contains far more data than the next decision needs. A search can return pages of text; code execution can produce logs, images or tables. Store the complete artefact in a file, object store or runtime state field. Give the model a stable reference, a small schema-aware excerpt and an operation for fetching more. This keeps bulky payloads out of every later prompt while preserving access.
The reference must carry more than a name. Include type, size, version and provenance, together with the permissions needed to read it. Retrieval through a reference should enforce the same access controls as the original tool. Expiry and deletion also need defined behaviour; a dangling handle can otherwise turn a recoverable context decision into a failed run.
Subagents provide another isolation boundary. Each can search or transform a large body of material in its own window, then return a compact finding to the orchestrator. Anthropic used this arrangement for parallel research and persisted the lead agent’s plan outside a truncating window. The trade-off is substantial token use and coordination. Subagents fit separable investigations better than tightly coupled work, and their summaries still need sources. Isolation moves token pressure; it does not establish correctness.
Test omissions and contradictions directly
A passing answer on a clean prompt says little about the context policy. Build fixtures around what the assembler removes and what it leaves in conflict. Place a necessary constraint late in history, inside a large tool result or across two records. Verify that the final prompt contains it, or that the agent can retrieve it when needed. Check the answer and the evidence path, not just the trace’s existence.
Contradiction tests need a specified resolution rule. Present an expired policy beside its replacement, or two tool observations with different timestamps and scopes. The expected behaviour may be to choose the authoritative current version, ask a question or report an unresolved conflict. Recency alone is not always authority. Preserve both records in state even when only one enters the prompt.
Vary position without changing the evidence set. Lost in the Middle found position-sensitive behaviour in several models and tasks, while some synthetic retrieval settings remained robust. Chroma’s later controlled report also found non-uniform degradation as inputs grew, with related distractors causing additional difficulty. Its focused conversational prompts outperformed full histories in the tested setting. These results justify local robustness tests; they do not define a universal curve or explain one causal mechanism.
Add compaction tests with a known evidence path. Compare the compressed run against a focused reference that contains the required facts, then follow references back to raw state. Test whether exact values survive, whether omitted facts trigger further retrieval and whether absent evidence produces an honest insufficiency response. Include missing, expired and unauthorised artefact references. Run the same cases after changing chunking, summary prompts, model versions or token thresholds.
Operate context as a controlled projection
Production telemetry should describe the projection, not merely count messages. Record token use by context component, retrieval candidates, included source versions, compaction events and artefact references. Keep the durable record subject to its own retention and privacy policy; tracing every prompt can create another sensitive store.
Use those records to locate waste and loss. Remove repeated tool feedback when it adds no evidence, but retain contradictory observations. Summarise settled phases while leaving active constraints exact. Isolate payloads that can be fetched on demand. Then rerun the omission and conflict suite before changing the policy.
A useful projection is recoverable: it contains enough authorised evidence for the next decision and states uncertainty when that evidence is missing. Neither the shortest prompt nor the fullest window guarantees that result. Context engineering succeeds when the system can explain both what the model saw and what it deliberately left outside.
Sources and further reading
- Liu et al., Lost in the Middle: How Language Models Use Long Contexts — controlled tests of evidence position and context length, v3, 20 November 2023.
- Packer et al., MemGPT: Towards LLMs as Operating Systems — main context, external storage, eviction and paginated retrieval, v2, 12 February 2024.
- Jiang et al., LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression — question-aware selection, token retention and recompression limits, v2, 12 August 2024.
- Anthropic, How we built our multi-agent research system — external plans, dynamic search and isolated subagent contexts, 13 June 2025.
- LangChain, Context Engineering for Agents — the write, select, compress and isolate framework for runtime context, 2 July 2025.
- Chroma, Context Rot: How Increasing Input Tokens Impacts LLM Performance — controlled length, distractor and focused-history experiments, 14 July 2025, with clarification of 16 July 2025.