When a language model answers from retrieved documents, it sometimes ignores what the documents say and falls back on facts it memorized during training. Research on this problem, called knowledge conflict, has mostly studied one swapped fact in one short passage. Real retrieved evidence usually forms a chain: document C mentions an entity that document B defines, and B relies on a fact stated in A. Because these models read strictly left to right, the order of the documents decides whether, at each mention, the model has already seen the contextual definition or has to fill the gap from memory. We propose to test a specific hypothesis. When a premise refers to an entity whose counterfactual definition has not yet appeared, the model binds that mention to its memorized fact. A later contradicting passage only partly undoes that binding. If this is right, ordering documents so that definitions come before references (a topological order of the dependency graph) should reduce memory fallback in multi-hop conflicts. It should do so beyond anything explained by simple recency.
Testing this cleanly requires an evaluation the field does not have. Static conflict benchmarks such as NQ-SWAP and ConFiQA are built on Natural Questions and Wikipedia, which have leaked into pretraining corpora. Dynamic, contamination-resistant generators such as DyVal use abstract symbols, where memory plays no role. We will build a generator of fresh multi-hop dependency chains over real, model-known entities. It will control four factors independently: document order, where in the chain the counterfactual edits sit, where distractor text sits, and how strongly the model believes the original fact. The generator is the enabling artifact. The ordering-driven fallback effect is the scientific claim.
Core Claim
In multi-document prompts whose premises form a directed acyclic graph (a dependency structure with no cycles), the rate at which a model reverts to memorized facts depends on whether each counterfactual premise is read before or after the premises that reference it. The claim goes further than "position matters." We predict three things. The penalty from non-topological order grows with how strongly the model believes the overridden fact. It disappears for matched fictional entities, which the model has no memory of. It persists when the total distance between each counterfactual premise and the question is held fixed. The first prediction follows from the entity-frequency stubbornness reported by Wang et al. and Song et al.. The third separates structural order from a recency effect.
Relation to Prior Work
Three lines of work motivate the gap. DyVal generates DAG reasoning problems in topological, reversed, or random description order and reports a 13.6% accuracy drop for GPT-4 on reachability under reversal. Its nodes are abstract, so it cannot say whether reversed order pushes models toward memory. Code work shows that definitions-first ordering helps in practice. Hierarchical Context Pruning places imported modules before the active file and raises CodeGemma-7B exact match from 36.69% to 41.21% while cutting prompts from about 50k to 8k tokens. RepoQA sorts files by global dependency before concatenation. Neither isolates ordering from pruning, and neither involves conflict with memorized facts.
Work on knowledge conflict supplies our measurement tools and our strongest competitors. Analysing the Residual Stream Under Knowledge Conflicts trains linear probes that detect conflict at about 90% accuracy around layer 14 of Llama3-8B and predict the final choice between context and memory at AUROC above 0.85 at layers 16–17. It does so only on short single-turn pairs. We reuse its probes to ask whether ordering shifts when conflict is detected. Jin et al. locate separate attention heads that retrieve memorized facts and heads that attend to prompt evidence. This gives a mechanistic target: under non-topological order, the memory heads should win at the early reference. ContextFocus adds a steering vector (a fixed direction added to intermediate activations) that pushes models toward context, evaluated on ConFiQA's multi-hop subsets. It never varies presentation order. DisentQA trains models to output separate contextual and parametric answers, raising counterfactual accuracy from 66.8% to 84.9%, but only on single passages.
Work on noisy retrieval constrains the design. Superposition Prompting processes each document in its own parallel branch and prunes irrelevant branches. It notes that this independent pruning struggles on multi-hop questions. That is exactly the setting where cross-document order matters, so we treat it as a baseline rather than a solution. Ext2Gen trains extract-then-answer behavior under up to 25 distractor chunks but treats chunks as unordered. Cho et al. show that models forge false connections as distractors accumulate. This motivates our test of distractors placed on the dependency path versus off it. On the Fragility of Benchmark Contamination Detection warns that perplexity and n-gram detectors are easy to evade. We therefore guarantee freshness by construction rather than detecting contamination after the fact.
Research Program
The generator samples a DAG of depth 2 to 5 whose nodes are real entities and whose edges are relations drawn from a knowledge base. Examples: "X was born in Y", "Y is the capital of Z", "Z's official language is W". The composed multi-hop questions are new even when each individual fact is well known. Before the main study, each fact is scored by closed-book prompting of the model under study, and only facts the model reliably recalls are kept. This recall confidence serves as the continuous "memory strength" variable. Counterfactual edits are placed at chosen nodes: at the root only, at a middle node only, or at several nodes, mirroring ConFiQA's split between one counterfactual and many. The edits propagate so that the context remains internally consistent.
Each edge becomes a short prose document. We also render a fictional twin of every item, with invented entity names and the same graph and wording, so that memory pull is zero. Distractor documents come in two kinds. On-path distractors mention chain entities with irrelevant relations. Off-path distractors belong to disconnected components. To confirm freshness, generated question–chain strings are checked for n-gram overlap against FineWeb.
The central manipulation compares four orders: topological, reversed, random, and a position-matched non-topological order. In the position-matched order, the counterfactual premises sit at the same token distances from the question as in the topological order, but the documents referencing them come before them. This condition is what separates structure from recency.
We measure three outcomes: contextual adherence (the answer follows the edited chain), memory fallback (the answer equals the original real-world composition), and chain breaks (the answer mixes the two). For mechanism, we run the existing conflict probes and memory/context head attributions at the reference token. We expect that under non-topological order, the probe at the first downstream reference will register memory, and that the late choice probe will more often favor memory.
If the effect holds, we test a cheap inference-time fix: reorder retrieved documents by a dependency graph built from which entities each document introduces and which it mentions, with no parser or training. We apply this to MuSiQue and to ConFiQA multi-hop items.
Baselines
The do-nothing reference is the retriever's own ranking order, or random order for generated items, with a standard answer-from-context instruction. Every method is compared against it on identical items.
The strongest existing intervention is ContextFocus steering, applied to Llama-3.1-8B-Instruct and a 70B model and fitted on the reported roughly 1.5k examples. Its paper reports ps well above unsteered models on ConFiQA. No numeric ps value appears on Lacuna, so we will rerun it rather than quote a figure. It is the strongest competitor because it targets memory fallback directly and was evaluated on multi-hop counterfactuals. We fit it once, with the same example budget, on randomly ordered training items, then evaluate it under every order. The question is whether reordering alone matches it and whether the two combine.
For noise robustness, the baseline is Ext2Gen. Its reported average on Llama3.1-8B-Instruct is 0.557 against 0.473 for the default model. We run its released model on our noisy items under each order. For conflict-aware prompting, we use DisentQA-style dual-answer prompting and Chain-of-Note (Zhang et al.), each given the same prompt-tuning budget of a handful of templates on a held-out split. Superposition Prompting is run on MuSiQue as the parallel-branch alternative.
The key ablation is the position-matched non-topological order, together with a "recency" order that puts all counterfactual documents last. Reordering beats the field if topological order lowers memory fallback on high-confidence facts by a margin larger than item-level bootstrap intervals, against both random order and the position-matched control. It must also add to steering rather than duplicate it.
Why not just…?
Why not just shuffle ConFiQA instead of building a generator? We will do this, as an external validity check. On its own, though, it cannot separate reading from recall. ConFiQA's chains come from Wikipedia-derived facts, so a model can reach the "original" answer through memorized question–answer co-occurrences rather than by binding each hop to memory. It also provides no fictional twin and no graded memory strength. The fragility of contamination detection means we cannot filter such items after the fact.
Why not just put the counterfactual documents last to exploit recency? This is the real threat. If recency order matches topological order, our structural claim fails, and the useful advice becomes "put conflicts last." The recency and position-matched conditions exist to settle this. We report it plainly if recency wins.
Why not just instruct the model or steer it? Instructions alone fell short of steering in ContextFocus. Cho et al. found that abstention instructions needed an added query-likelihood rescoring to be effective. Steering is a fixed global push and may overcorrect toward gullibility on genuinely noisy context (Hong et al.). Reordering costs nothing and is applied per item. The combination experiment shows whether the two are redundant.
Why not just use fictional entities, as DyVal does with symbols? Fictional entities remove the very force under study. We keep them as the zero-memory control. The prediction is an interaction: an order penalty that grows with memory strength and vanishes on the twins.
Proposed Experiments and Decisive Checks
The main study covers Llama2-7B and Llama3-8B, whose conflict probes already exist, plus Llama-3.1-8B-Instruct and a 70B model. Each cell in the grid of order × edit placement × distractor type × memory-strength tertile gets 500 generated items, matching DyVal's per-condition scale. External checks use shuffled ConFiQA multi-hop items, NQ-SWAP, and MuSiQue.
The decisive result has three parts. First, a positive interaction between non-topological order and memory strength on memory fallback. Second, a near-zero order effect on the fictional twins. Third, a surviving gap between topological and position-matched orders. Mechanistically, we expect the layer-14 conflict probe to flag memory at the first downstream reference only under non-topological order.
For distractors, we predict that on-path distractors mostly raise chain breaks and off-path distractors mostly dilute accuracy uniformly. A null result is still informative: if reversed order hurts equally on fictional twins, then DyVal's reversal drop reflects difficulty composing out of order, not memory fallback, and the conflict literature can safely ignore order.
References
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- A Causal View of Entity Bias in (Large) Language Models
- Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded Attributions and Learning to Refuse
- Hierarchical Context Pruning: Optimizing Real-World Code Completion with Repository-Level Pretrained Code LLMs
- RepoQA: Evaluating Long Context Code Understanding
- Analysing the Residual Stream of Language Models Under Knowledge Conflicts
- Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Models
- ContextFocus: Activation Steering for Contextual Faithfulness in Large Language Models
- DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answering
- Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generation
- Aligning Extraction and Generation for Robust Retrieval-Augmented Generation
- Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Models
- Why So Gullible? Enhancing the Robustness of Retrieval-Augmented Models against Counterfactual Noise