Systems that answer multi-step questions by searching repeatedly have a built-in contradiction. Each search round adds paragraphs to the model's context so that later steps can find the facts that link one step to the next. Every paragraph that turns out to be irrelevant also makes the model more likely to be misled. We propose a test that works with any text-only model API. After each search round, ask the model to state its current intermediate conclusion under a few reworded versions of the same instruction. If the conclusion changes depending on the wording, the context is not supporting it. Removing a paragraph and checking whether the conclusion becomes more stable shows whether that paragraph is evidence or a distractor.
The same signal handles a second failure. Sometimes the corpus simply lacks the fact needed for the next step, much as knowledge bases used to label training data are missing most facts about most entities. In that case no paragraph stabilizes the conclusion, and the right action is to search differently or to say "unanswerable" rather than build a chain from whatever is in the context. The model's sensitivity to how it is instructed, usually treated as a nuisance, becomes a cheap per-step measure of whether the retrieved evidence is doing its job.
Core Claim
In iterative retrieval-augmented question answering, a set of instruction paraphrases yields a measure of how consistent the model's intermediate step is. "Iterative retrieval" here means the model alternates between writing a reasoning step and using that step as a search query. "Intermediate step" means the bridge entity or partial answer at hop $k$. We claim three things for this consistency measure. First, it separates supporting from distracting paragraphs better than asking the model whether each paragraph is relevant. Second, it separates "evidence present but buried" from "evidence absent from the corpus." Third, using it to remove paragraphs and to abstain improves answer F1 and selective-prediction quality over context accumulation at a matched number of model calls. No existing method does this. The claim is falsifiable by one ablation: paraphrase-based stability must beat both single-prompt leave-one-out and temperature-sampling consistency at the same call budget. If it does not, the reframing reduces to known techniques and we will report that.
What Prior Work Establishes and Leaves Open
Three papers motivate the problem by showing that iterative retrieval works but accumulates context without cleaning it.
- IRCoT uses each chain-of-thought sentence as the next search query. It raises recall by 11–21 points and F1 by up to 15 on HotpotQA, 2WikiMultiHopQA, MuSiQue and IIRC. It appends every retrieved paragraph and never removes earlier ones.
- ITER-RETGEN loops whole draft answers back into retrieval. It likewise trusts the model to ignore noise in later rounds.
- DRAGIN decides when to retrieve from token entropy and attention. It reaches 0.3931 F1 on 2WikiMultihopQA with LLaMA2-13B, against 0.3076 for FLARE. It needs internal model access and does not prune context.
Our method differs from all three in that it removes context, and from DRAGIN in that it uses only generated text.
A second group shows the cost of noise and supplies the strongest noise-handling methods, but only for flat, single-step contexts.
- Cho et al. show zero-shot readers collapse as retrieved documents grow. Their fix (DAS) combines an "unanswerable" escape instruction with query-likelihood reweighting, giving a 64% average EM gain on single-hop open-domain QA.
- Ext2Gen trains a model with preference optimization to extract supporting sentences before answering. On Llama3.1-8b-instruct it lifts its benchmark score from 0.473 to 0.557, with up to 25 distractor chunks, and it beats separate compressors.
- UDA and Dhruva et al. document how buried targets and excess context force false conclusions.
None of these methods acts across hops. An extraction error at hop $k$ becomes a bad query at hop $k+1$, and that is exactly the regime we target.
The method itself comes from the prompt-sensitivity literature.
- Chen et al. show that prediction disagreement across instruction paraphrases correlates negatively with accuracy (Pearson $r$ up to −0.40). Their SENSEL method uses this disagreement to abstain, gaining +4.1 points of area under the F1–coverage curve on GPT-J-6B. They evaluate only single-step classification.
- ProSA links sensitivity to flat output distributions and finds it peaks on complex reasoning. This suggests the signal should be strongest in multi-hop settings.
We move SENSEL's signal from the final answer to each intermediate hop, and from abstention alone to deciding which paragraphs to keep.
The knowledge-base incompleteness literature plays two roles.
- Constraint on design. Min et al. show Freebase lacks birthplace for 93.8% of people, and that false negatives outnumber positives 3–10×. In single-sentence extraction they model missing facts as latent variables rather than as negatives, and Peng et al. do the same with positive-unlabeled learning. By analogy, a hop whose bridge fact is missing should trigger re-search or abstention, not a forced join.
- Evaluation lens. 2WikiMultiHopQA is built from Wikidata rules and acknowledges mismatches between text and triples. Li et al. show that evaluation sets derived from incomplete knowledge bases penalize correct predictions. So "non-gold" paragraphs must be treated as unlabeled, not as known distractors, when we score pruning.
MuSiQue supplies unanswerable contrast questions (MuSiQue-Full), which give a natural test of missing evidence.
Research Program
Phase 1: measure the cascade. Take IRCoT as the base loop and run it under 12 semantically equivalent instruction paraphrases, built from ProSA's template families. We measure how often the hop-1 query changes, how much the overlap between retrieved paragraphs drops, and how far final F1 varies. This tests whether instruction brittleness actually propagates through retrieval into contextual distraction, which no file paper has quantified. If variance from instruction wording is negligible for multi-hop pipelines, the remaining phases lose their premise, so this phase is a gate.
Phase 2: the hop-stability signal. At hop $k$, with context bank $C$, we query the model with $J=4$ instruction paraphrases. From each response we extract the intermediate statement's bridge entity or partial answer $e^{(j)}$. Stability is defined as
$$A(C) = \frac{2}{J(J-1)} \sum_{i<j} \mathbb{1}\left[e^{(i)} \equiv e^{(j)}\right],$$
where $\equiv$ is normalized string match after entity canonicalization.
When $A(C)$ falls below a threshold tuned on a development split, we compute the stability gain $A(C \setminus p) - A(C)$ for each paragraph $p$ in the bank. The bank is capped at a fixed size, so this costs at most a bounded number of extra calls. Paragraphs whose removal raises stability are removed from the bank. Removal is conditional: a removed paragraph is restored if a later hop's query retrieves it again. This guards against discarding a paragraph that only becomes relevant later.
Phase 3: missing-evidence handling. If no single removal restores stability, and stability stays low after one reformulated search (we re-query with the paraphrases' disagreeing entities as alternative queries), the hop is flagged as "unsupported." The system then either returns "unanswerable" or reports a confidence-ranked partial answer.
To create controlled incompleteness, we delete the gold bridge paragraph from the corpus for a random subset of 2WikiMultiHopQA and MuSiQue-Ans questions. This mimics the missing-fact rates Min et al. report. We test whether the system abstains instead of hallucinating, and whether it ever recovers through an alternate path that the corpus does support.
Every phase runs with open-weight models (Llama3.1-8B-instruct and Llama 3-70B-Instruct) treated as text-only, so that DRAGIN can also be run on the same weights. The 8B model matters because the file's open problems flag small models as most overwhelmed by long context.
Baselines
Naive accumulation. The reference point is plain IRCoT with full accumulation, run with identical retriever, corpus, top-$k$, hop limit and few-shot demonstrations. We do not have per-dataset IRCoT numbers from the file (no reported result found on Lacuna for IRCoT per-dataset F1). We therefore reproduce it and treat our reproduction as the anchor. Any gain must be measured against it at equal hop limits.
Strongest existing dynamic method. This is DRAGIN, which uses internal entropy and attention to decide when and what to retrieve. It is the closest competitor that adapts per step, and its reported 2WikiMultihopQA F1 of 0.3931 with LLaMA2-13B (DRAGIN) is the number to beat on that benchmark. We rerun DRAGIN on our models with its released settings. Beating it means higher F1 on 2WikiMultihopQA and MuSiQue with fewer or equal context tokens, while using no internal model access.
Strongest noise-robust reader. This is Ext2Gen applied inside the IRCoT loop at every hop: extract supporting sentences, then reason. Its reported flat-context score is 0.557 versus 0.473 on Llama3.1-8b-instruct (Ext2Gen). It represents the best learned answer to distraction. Ext2Gen is trained while our method is not, so we report it both as released and with our method layered on top. If our pruning adds nothing on top of Ext2Gen extraction, that is a negative result we will state.
Abstention baseline. For abstention we compare against DAS-style escape instructions (Cho et al.) and against SENSEL applied only to the final answer, scored by area under the F1–coverage curve. For SENSEL the file reports only its single-step +4.1 gain and no multi-hop number.
Key ablations. These isolate the paraphrase ingredient at matched model-call budgets. The first is single-prompt leave-one-out, which removes a paragraph and checks whether the greedy intermediate answer changes. The second replaces paraphrases with $J$ temperature samples under one instruction (self-consistency). The third is a model-judged relevance prompt per paragraph.
Fairness rules. Every method gets the same corpus, retriever, hop cap, demonstration set and development-split tuning budget. All results are plotted against total model calls and total context tokens, never against hops alone.
Why not just…?
Why not just ask the model whether each paragraph is relevant? This is the cheapest competitor and a real threat. It costs one call per paragraph and needs no paraphrases. Its weakness, suggested by Dhruva et al. and Cho et al., is that models exposed to distractors make forced connections. A model that has already been misled will plausibly also judge the misleading paragraph relevant. The relevance-prompt ablation settles this directly. If it matches our method on pruning precision against audited labels and on final F1, we will adopt it and the paper becomes an analysis of the cascade.
Why not just sample several times at temperature instead of rewording the instruction? Sampling also produces a consistency signal. ProSA ties instruction sensitivity to flat output distributions, which sampling also detects. Paraphrase variation may therefore add nothing. Our argument is that rewording probes a different axis: whether the conclusion depends on how the model is framed, rather than on decoding randomness. Chen et al. found instruction-level sensitivity informative beyond output-probability baselines. That is evidence from single-step classification only, so the matched-budget sampling ablation is decisive here.
Why not just cap the context with a re-ranker or a separate compressor? Ext2Gen already reports that compressors (Recomp, CompAct, EXIT) underperform in-generation extraction. Zhang et al. (Coarse-to-Fine Knowledge Selection) note that pre-filtering strips relational cues. In multi-hop settings, a paragraph's relevance depends on a bridge entity the re-ranker never saw at query time. We still run a cross-encoder top-$m$ cap inside the loop as a baseline, matched on context tokens.
Why not just use a bigger model? IRCoT shows a 3B model with iterative retrieval beating a 175B model with one-step retrieval. ProSA reports that GPT-4o can be more sensitive to instructions than Llama 3-70B. Scale does not remove either problem. Running both 8B and 70B models will show whether the gain shrinks with scale, which is itself a reportable finding.
Proposed Experiments and Decisive Checks
Benchmarks and metrics. - HotpotQA (full-wiki), 2WikiMultiHopQA, MuSiQue-Ans and IIRC for answer EM and F1. Human references give scale: 91.40 F1 on HotpotQA (HotpotQA) and 78.0 answer F1 on MuSiQue (MuSiQue). - MuSiQue-Full and our bridge-deletion variants for abstention, measured by F1–coverage area and by the rate of hallucinated answers on questions with deleted evidence. - Retained-bank precision and recall against gold supporting paragraphs. We treat this as a lower bound, and manually audit 200 "wrongly kept" paragraphs to count valid unlabeled evidence. This applies the false-negative lesson of Li et al. and Ho et al. - Context tokens and model calls per question for every system. - Phase 1 reports F1 variance across the 12 paraphrases, so pipeline robustness to instruction wording is also an outcome.
Decisive checks. Three results would confirm the proposal:
- At an equal call budget, paraphrase stability beats single-prompt leave-one-out, temperature-sampling consistency and relevance prompting on retained-bank precision and on final F1 for MuSiQue. MuSiQue is the hardest of the four and the one built to resist shortcuts.
- On bridge-deleted questions, the "unsupported hop" flag cuts hallucinated answers substantially relative to IRCoT and DAS, without lowering F1 on intact questions.
- The method beats DRAGIN's reported 0.3931 on 2WikiMultihopQA under a matched model, using text access only.
Two results would refute it. If the sampling or leave-one-out ablations match our method, the paraphrase ingredient is unnecessary. If Phase 1 shows little variance across instruction wordings, the cascade is too small to exploit. Either outcome is informative, and we would report it as the main result.
References
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
- Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy
- DRAGIN: Dynamic Retrieval Augmented Generation based on the Real-time Information Needs of Large Language Models
- Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering
- Aligning Extraction and Generation for Robust Retrieval-Augmented Generation
- UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-World Document Analysis
- Solving Open-Domain Multiple Choice Questions with Textual Entailment and Text Similarity Measures
- On the Relation between Sensitivity and Accuracy in In-Context Learning
- ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs
- Distant Supervision for Relation Extraction with an Incomplete Knowledge Base
- Distantly Supervised Named Entity Recognition using Positive-Unlabeled Learning
- Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps
- Active Testing: An Unbiased Evaluation Method for Distantly Supervised Relation Extraction
- ♫ MuSiQue: Multihop Questions via Single-hop Question Composition
- Coarse-To-Fine Knowledge Selection for Document Grounded Dialogs
- HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering