Paraphrase-Stable Search for Multi-Hop Agents Moving into New Domains

Systems that answer multi-step questions by searching, reading, and searching again (for example "where was the director of this film born?") write their own intermediate search queries. If two users ask the same question in different words, these agents often write different intermediate queries, retrieve different documents, and reach different answers. One wrong document at an early step misdirects every later step. We propose to measure this sensitivity hop by hop, and to train it away with a simple signal. Two phrasings of the same question should lead the agent to the same evidence at every step. The test bed is the setting where the problem should be worst: taking an agent trained on general Wikipedia questions and adapting it, in stages, to biomedical literature, whose vocabulary is far from what it learned to search with.

The opportunity is that current work treats three things separately: wording variation, distracting retrieved text, and domain adaptation. In iterative search they form one causal chain. A domain shift changes how the agent phrases its sub-queries. Unusual phrasing pulls in plausible but irrelevant documents. Those documents then hijack the next hop. If that chain is real, it can be cut at its first link, before any distractor enters the context window, rather than by filtering noise after the fact.

Core Claim

We claim that much of a multi-hop agent's accuracy loss after domain transfer comes from phrasing instability at intermediate hops. Concretely, the evidence the agent retrieves at hop $k$ changes when the question is reworded, even though the needed evidence does not. We further claim that a training reward defined on retrieved evidence, rather than on sub-query text or final answers, reduces this instability and the distractor cascades that follow from it. This reward is applied inside a staged general-to-domain curriculum. It should beat stronger reasoning chains and post-hoc distractor filters on paraphrased domain questions, without erasing the agent's general search behavior.

Let $q$ be a question and $q'$ a validated paraphrase of it. Let $E_k(q)$ be the set of passages the agent retrieves at hop $k$ when run on $q$, and let $K$ be the number of hops. The quantity we measure and reward is the mean per-hop evidence agreement

$$ C(q,q') = \frac{1}{K}\sum_{k=1}^{K} \frac{|E_k(q)\cap E_k(q')|}{|E_k(q)\cup E_k(q')|}, $$

gated by answer correctness, so that agreeing on wrong evidence earns nothing. Retrieval is not differentiable, so $C$ enters training as a preference signal. Pairs of sampled reasoning chains are ranked by correctness and then by $C$, and these pairs are used with Direct Preference Optimization, a method that fine-tunes a model directly from "this output is better than that one" pairs.

Relation to Prior Work

Iterative retrieval methods supply our base agent and our main competitors. IRCoT alternates a reasoning step with a search that uses that step as its query. CoRAG fine-tunes models on rejection-sampled chains of sub-queries and sub-answers. EVO-RAG trains a search agent with a staged reward over search steps. All three train and test on general Wikipedia benchmarks with the questions as written. None measures whether a reworded question yields the same trajectory, and none adapts the agent across domains. EVO-RAG's curriculum runs across steps within one task, not across domains. CoRAG also names cascading errors from one bad hop as an open problem, which motivates us to look at the earliest hop where a cascade begins.

Two papers supply the consistency idea, and we change what it is applied to. Weakly Supervised Multi-task Learning for Semantic Parsing forces paraphrase pairs to produce the same logical form through a divergence penalty. PARA4QA learns which paraphrases help by supervising them with downstream answers. Both work on single-shot parsing or lookup over a knowledge graph. We impose consistency on the evidence retrieved at each hop of a free-text search loop, a target neither could define. PARA4QA's finding that general-purpose paraphrasers underperform task-supervised ones also constrains our design: we do not canonicalize queries with an external paraphraser.

Distractor-handling work defines the competing remedy. Ext2Gen trains a model to extract relevant sentences before answering. Distraction-aware Answer Selection uses abstention prompts and query-likelihood rescoring. Reason-KE teaches an explicit "apply or ignore" step for each fact and loses only 6.3% accuracy under four distractors on MQuAKE-CF. All three act after retrieval. Our proposal acts before it. Combining the two families is a fair experiment, not an assumption. MergePRAG avoids context bloat by compiling passages into weight updates. As the file notes, it would bake a retrieved distractor into the weights, which makes upstream retrieval stability matter even more there. We treat it as a reference point, not a component.

The staged-adaptation literature supplies the curriculum. Training moves from distant data to near data (Neural Machine Translation Training in a Multi-Domain Scenario, QCRI's IWSLT'16 systems). Domain pretraining is separated from task pretraining (Don't Stop Pretraining). General instruction tuning buffers narrow domain tuning (TALLRec, MoZIP). These results come from single-turn tasks. Whether such staging preserves a search policy (when to search, how to phrase queries, when to stop) is untested, and that is part of what we measure.

Research Program

The first phase is diagnostic and could stand alone as a result. We build paraphrase sets for multi-hop questions in two domains. In the general domain, we use MuSiQue, whose questions are composed from single-hop parts, so that each part can be reworded independently. In the biomedical domain, we use QAngaroo MedHop, whose templated drug-interaction queries we rewrite into several natural-language phrasings. These range from lay wording to pharmacological wording, are generated by an LLM, and are filtered by human check for meaning preservation. We enlarge MedHop's support sets with extra Medline abstracts so that runs exceed the 50-document range where MedKGQA reports sharp degradation. We then run a CoRAG-style agent on every paraphrase, record $E_k$ at each hop, and ask three questions. Does per-hop agreement fall more after domain transfer than final accuracy alone would suggest? Is the first hop with low agreement predictive of final error? Do the documents that enter only under some phrasings score as distractors, in the sense of being retrieved but never used by gold reasoning?

The second phase builds the method. We start from an open 7–8B model trained as a general multi-hop agent on MuSiQue and HotpotQA. We then apply three stages. First, we continue pretraining on biomedical abstracts. Second, we train on general multi-hop chains, replaying the search behavior learned earlier. Third, we train on domain multi-hop chains with the evidence-agreement preference signal. Paraphrases are generated on the fly for training questions only; test paraphrases are held out and written with different templates and annotators.

Baselines

The naive reference is single-pass retrieval with a large top-$k$ and the same reader. It shows the floor. On MuSiQue, CoRAG reports single-step retrieval recall of 29.0% against 47.1% for its iterative chains.

The strongest existing method is CoRAG itself, retrained on the same backbone with the same curriculum stages, data, rejection-sampling budget, and number of preference pairs. The only difference is that its preferences use correctness alone. This makes it the exact proposal minus the evidence-agreement signal. Beating it means higher accuracy and higher per-hop agreement on held-out paraphrases of MedHop, with general-domain MuSiQue F1 no worse than CoRAG's. For context on MedHop, the best reported accuracy in the file is 64.8% for MedKGQA, against 85.0% human accuracy reported in the MedHop dataset paper. These are on original queries, not paraphrases, and we report our numbers on both. We also run IRCoT and EVO-RAG under matched retrieval budgets.

The post-retrieval competitor is CoRAG with an Ext2Gen-style extraction step trained on the same data. On its own benchmark, Ext2Gen-R2 reports an average score of 0.557 against 0.473 for the default Llama3.1-8B-instruct. A Reason-KE-style apply/ignore chain serves as a second filter variant.

Three ablations isolate the key ingredient. The first applies consistency to final answers only, as in WS-MTL, transplanted to retrieval QA. The second applies consistency to sub-query text similarity. The third uses the evidence signal without the staged curriculum, with all data mixed in one stage. If answer-level consistency matches evidence-level consistency, our central mechanism is wrong.

Why not just…?

Why not just domain-adapt the retriever? This is the most serious threat. A retriever pretrained on biomedical text might absorb vocabulary mismatch so that phrasing no longer matters. We do not argue it away. Every condition is run with both a general and a domain-adapted retriever. If the domain retriever alone restores per-hop agreement and accuracy, the right conclusion is that the problem lives in the retriever, and the diagnostic phase still stands as the finding. We expect a residual gap for a specific reason. Paraphrases change which bridge entity the agent names in its next sub-query, not just the words used, and no retriever can recover an entity the agent did not ask about. We test this by labeling disagreements as lexical or entity-level.

Why not just canonicalize questions with an off-the-shelf paraphraser? PARA4QA found that general paraphrase models trained on corpora like Quora underperform answer-supervised selection, partly because they blur technical terms. Canonicalizing the user question would also not touch the agent's own later sub-queries, where we hypothesize the drift happens. We include a canonicalize-then-search baseline to show this directly.

Why not just filter distractors after retrieval? Ext2Gen, DAS, and Reason-KE show that filtering works when the needed evidence is present among the distractors. A phrasing-induced miss at hop one means the needed passage was never retrieved, and no filter can restore it. The filter baseline above, together with gold-evidence recall at each hop, separates "evidence retrieved but misused" from "evidence never retrieved." If filtering closes most of the gap, the cascade story is weaker than we claim.

Why not just train on a single mixture of general and domain data? Prior staged-training results (IWSLT'16, Don't Stop Pretraining) favor staging, but none concerns search policies. The single-mixture ablation tests this claim instead of assuming it. It reports domain accuracy, general MuSiQue and HotpotQA F1, average hops, and the rate of stopping early.

Decisive Checks

The idea fails in any of three cases: per-hop evidence disagreement under paraphrase does not predict final errors on MedHop, a domain-adapted retriever eliminates the disagreement on its own, or answer-level consistency matches evidence-level consistency. The idea succeeds if three conditions all hold under a fixed retrieval budget. Evidence-level preferences must raise both paraphrase-averaged MedHop accuracy and per-hop agreement over the matched CoRAG baseline. The gain must persist with the domain retriever and with the extraction filter added. General-domain multi-hop F1 must stay within noise of the general agent. We report the spread of accuracy across paraphrases of each question, alongside its mean, as the main robustness metric.