Deferred Joins: Lazy Evidence Pruning for Multi-Hop Retrieval

Language models answering questions from retrieved documents get worse when the prompt is padded with irrelevant text. Accuracy can collapse under well-chosen distractors, and stuffing in more passages often lowers accuracy rather than raising it. The obvious fix is to score each retrieved document on its own and throw away the low scorers before generation. That fix breaks multi-hop questions. In a question like "Where was the director of film X born?", the document about the director's birthplace looks irrelevant to the question until you already know who the director is. It gets pruned, and the reasoning chain is gone.

We propose to treat pruning the way a database treats an expensive join: defer it until its dependencies are known. Every retrieved document is encoded once and cached in a form that does not depend on where it will sit in the prompt. Low-scoring documents are not discarded. They stay in a cheap pool that has been encoded but not yet placed into the prompt. After the model extracts an intermediate "bridge" fact from the documents it kept, it re-scores the pooled documents conditioned on that fact. Because the documents are already cached, each re-score costs only the few tokens of the bridge and the question, not the document. The model reads only the documents that survive this conditional check, so the prompt stays small, and the number of sequential steps grows with the number of hops rather than the number of documents.

Core Claim

Isolated-saliency pruning fails in a specific, measurable way: on bridge-type multi-hop questions, the second-hop gold document ranks low when scored against the question alone. Deferring the prune decision and conditioning it on extracted bridge facts can recover those documents. On a position-agnostic cache, this re-scoring is nearly free. The result should be higher answer F1 than isolated pruning on bridge questions. It should also need fewer materialized tokens and fewer sequential model calls than stepwise compression. On single-hop questions and comparison questions, where each gold document is relevant by itself, there should be no gain and no harm.

Where This Sits

The work borrows a mechanism and a substrate from two papers and sharpens a weakness in one. Superposition Prompting supplies the mechanism. It processes each document in parallel with the query, computes a saliency score per path, and prunes low-saliency paths before generation. It reports a 43% accuracy gain over naive concatenation on NaturalQuestions-Open with MPT-7B. Its pruning is one-shot and isolated: a path is judged without reference to any other path. We keep its parallel structure and replace that single prune with a deferred, conditional one. LazyAttention supplies the substrate. It stores key-value caches without positional rotation and applies rotary position encoding inside the attention kernel, so a cached document can be placed anywhere in a prompt at about 0.2% runtime overhead. That paper makes no decision about which documents are noise. We use its cache so that pooled documents can be re-scored and later placed without re-encoding.

The motivating evidence comes from the distraction literature. Breaking Focus shows accuracy drops averaging over 45% under coherent distractors, and shows that "ignore irrelevant text" instructions do not help. LDAR documents the inverted-U curve, where a learned band of passages beats the full context while using 25–63% of the tokens. MuSiQue was built so that single-hop shortcuts fail, which is exactly where isolated scoring should hurt. One constraint shapes the design. CompAct also defers reading: it walks documents in segments of five and stops early, gaining 7.0 F1 over RECOMP on HotpotQA. Its decisions are greedy and sequential, though, so its latency grows with the number of segments. Our rounds instead run in parallel across all documents, one round per hop. A second constraint comes from DAS. It shows that scoring a document by the likelihood of the query given that document is a strong distraction filter, and that an explicit abstain option matters. We reuse both ideas.

Research Program

The system runs in rounds. Let $D$ be the retrieved documents, each cached position-agnostically, and let $q$ be the question.

Round 0. Every document $d$ gets an isolated score $s_0(d) = \frac{1}{|q|}\log p(q \mid d)$. This is the average per-token log-likelihood of the question when the question follows the cached document. The top few documents are kept, and the rest go to the deferred pool.

Each later round. The model reads only the kept set and emits a short bridge statement $b_t$, such as "The director of X is Y." It may instead declare that the evidence is complete or that the question is unanswerable. Each pooled document is then re-scored as

$$s_t(d) = \frac{1}{|q|}\log p(q \mid d,\, b_t),$$

where $b_t$ and $q$ are appended after the cached document. This prefill costs $|b_t| + |q|$ tokens per document and runs as one batched pass. Documents whose conditional score rises above a threshold are promoted into the kept set. The loop stops when the model declares completion or when a hop budget runs out; we use four hops, matching MuSiQue's maximum.

Guarding against a bad bridge. A hallucinated bridge would steer re-scoring toward the wrong documents. To limit this, the system keeps the top two or three bridge candidates sampled from the kept set and takes, for each document, its maximum score across candidates. It also retains DAS-style abstention when no document rises.

What is built. The engineering artifact is a batched conditional-scoring kernel over a position-agnostic cache. We will build it on the LazyAttention approach for RoPE models.

Analysis before any accuracy claims. We first verify the motivating failure. Using the gold supporting-fact labels in HotpotQA and MuSiQue, we measure the rank of each gold document under $s_0$, separated by hop position and question type. If second-hop gold documents do not rank systematically low under isolated scoring, the premise is wrong, and the project reduces to a reranking tweak.

Baselines

The naive reference point is plain concatenation of all retrieved documents into one long prompt. On HotpotQA distractor this means two gold and eight distractor paragraphs. On MuSiQue it means the provided paragraph set. This baseline shows how much distraction costs a modern reader. The only reported number in the file for this setting is the original BiDAF result, 58.28 answer F1 against human 91.40 on HotpotQA distractor (HotpotQA). It is a historical anchor, not a target, since all systems here are rerun with the same LLM.

The strongest direct competitor is Superposition Prompting. It shares our parallel structure and is the method we claim to fix. The file reports only that on MuSiQue it was "often the only method capable of achieving non-zero scores on the most difficult multi-hop queries." No reported F1 for Superposition Prompting on HotpotQA or MuSiQue was found on Lacuna, so we will reproduce it. It will run on the same model, retriever output, and cache. Its pruning threshold will be tuned on the same development split with the same number of tuning trials as ours. Beating it means higher answer F1 on MuSiQue 3- and 4-hop and on HotpotQA bridge questions, at equal or lower materialized tokens, with no loss on comparison questions.

The strongest sequential competitor is CompAct, which reports 7.0 F1 over RECOMP on HotpotQA. It runs with the same reader. We compare at matched answer F1 and report sequential model calls and wall-clock time to first answer token. Beating it means matching its F1 with fewer sequential rounds, or exceeding it at equal latency.

We also include Ext2Gen, an extract-then-answer method trained with preference optimization. It reports 0.557 average QA score against 0.473 for the default Llama3.1-8b-instruct. We use the same 8B model so that it serves as a trained-model reference. Our method is training-free, so matching Ext2Gen would already be meaningful.

Two ablations isolate the key ingredient. The first, "isolated multi-round," runs the same rounds but re-scores pooled documents with $s_0$, without the bridge. It separates the value of conditioning from the value of simply taking more rounds. The second, "eager conditional," re-encodes each pooled document from scratch under the bridge instead of using the cache. It should give the same accuracy at much higher prefill cost, which isolates the systems contribution.

Why not just…?

Why not just lower Superposition Prompting's pruning threshold so bridge documents survive? Keeping more paths reduces pruning loss but brings distractors back. That is the inverted-U tradeoff LDAR documents. We will sweep the threshold and plot answer F1 against materialized tokens. If any point on that curve matches deferred joins at equal tokens, our method adds nothing. The round-0 rank analysis predicts that it will not: if second-hop gold documents sit below many distractors under isolated scores, no threshold can admit them without also admitting those distractors.

Why not just re-retrieve with the bridge as a new query, or rerank the pool with a cross-encoder on question plus bridge? This is the most serious threat. It captures the same conditioning idea with off-the-shelf parts. We will implement it inside the same round structure, using a separate retriever or cross-encoder in place of $s_t$. If it matches our accuracy, the honest conclusion is that conditional deferral is the contribution and the reader's own likelihood is not needed. The remaining claim would then be cost and consistency: our score reuses the reader's cached states and reflects the reader's own uncertainty, which LDAR suggests is model-specific. We will report both outcomes.

Why not just give everything to a long-context model? Ten paragraphs fit easily in modern context windows. Breaking Focus and Help or Hurdle? report large accuracy losses from added context: over 45% average under adversarial distraction, and 9.5% average from tool context. The concatenation baseline measures this directly on our benchmarks. If concatenation matches us on MuSiQue, distraction is not the bottleneck there, and the gain would have to come from cost alone.

Why not just use CompAct's stepwise compression? CompAct already defers reading. Its cost is rounds proportional to document count, and its greedy stopping can end before a later segment supplies the second hop. The matched-F1 latency comparison settles which approach is better.

Proposed Experiments

The main evaluation uses HotpotQA distractor and MuSiQue-Ans, reporting answer F1 and EM. We also report support recall of the materialized set against gold supporting facts, materialized tokens, sequential rounds, and time to first token. On MuSiQue the results are separated by 2-, 3-, and 4-hop. MuSiQue-Full tests whether conditional re-scoring correctly fails to promote anything on unanswerable contrast questions; the risk is that a confident wrong bridge pulls in a plausible distractor. NaturalQuestions-Open is a single-hop regression check. There the method should stop after round 0 and match Superposition Prompting. Models are open RoPE models around 8B parameters, so that LazyAttention-style kernels apply.

Three checks decide the question. First, the round-0 analysis must show low isolated ranks for second-hop gold documents on bridge questions. Second, the gain over Superposition Prompting must appear on bridge questions and vanish on HotpotQA comparison questions, a built-in negative control. Third, the isolated multi-round ablation must fall short of the full method, so that conditioning rather than extra rounds is shown to do the work. If the bridge-query re-retrieval baseline ties the full method, we report the result as evidence for deferral as a principle rather than for reader-likelihood scoring.