Minimal Pairs at the Bridge Hop: Testing Whether Biomedical Retrievers Follow the Chain or the Words

Multi-hop question answering asks a system to chain facts. For example, it might first find which drug is contraindicated for a condition, then find which protein that drug targets. Retrieval systems tend to fetch documents that share words with the question, so a natural failure is that the second hop retrieves evidence for the wrong first hop. The wrong evidence is retrieved because it looks almost identical on the surface. We propose to measure this failure directly with hop-level minimal pairs. These are two multi-hop questions that differ by one word in the first step, such as "indicated" versus "contraindicated", or "agonist" versus "antagonist". The flip changes the intermediate "bridge" entity and therefore the final answer, while leaving more than 90% of the words shared. If a retriever is following the reasoning chain, its second-hop retrieval should change when the first hop changes. If it is matching words, the second-hop retrieval will stay put.

We place this test in biomedicine for two reasons. Relational opposites there are common, consequential, and lexically close. Domain pre-training, meaning continued training of a language model on domain text such as PubMed, is the default recipe there, and it is unknown whether that recipe helps or hurts this specific failure. A model that has absorbed dense drug–disease co-occurrence statistics might become better at telling a contraindication from an indication. It might instead become more inclined to retrieve whatever usually co-occurs with the drug name. The note proposes a benchmark that separates these two possibilities. It also proposes a training fix whose central ingredient can be isolated by ablation: contrastive negatives built from the minimally edited path, applied to the retriever's state after the first hop rather than to the question alone.

Core Claim

We hypothesize that in iterative dense retrieval, lexical-overlap errors propagate along the chain. A retriever that encodes "question plus retrieved hop-1 evidence" inherits the hop-1 evidence of the more frequent, lexically matching path. The fix that works for single-hop retrieval only partly transfers. That fix is penalizing similarity between a question and its minimally edited twin, introduced by Exploring Contrast Consistency of Open-Domain Question Answering Systems on Minimally Edited Questions. It does not fully carry over to multi-hop retrieval, because the error is reintroduced at the second hop through the retrieved context.

The proposal makes two testable claims. First, contrast consistency at hop 2 is substantially worse than at hop 1 for every retriever family, and the gap is larger for domain-pretrained encoders on co-occurrence-heavy relations. This is the claim most likely to be false, and the most interesting part. Second, adding minimal-pair negatives at the post-hop state closes most of that gap. Generic hard negatives, or question-level minimal pairs alone, do not.

How This Sits Against Prior Work

Three papers supply the evidence that motivates the work. MuSiQue builds multi-hop questions by composing single-hop ones and filtering out questions answerable without the first hop. It shows that earlier benchmarks are solvable by "disconnected reasoning", that is, skipping hops. Its contrast questions, however, remove whole supporting paragraphs. We instead edit a single relation word inside a hop. It Is Not Easy To Detect Paraphrases (SemAntoNeg) tests sentence encoders on negation and antonym triplets. It finds that a widely used general encoder scores 1.9% against a 33.3% random baseline, while NLI-trained models reach 93.6%. That work covers single sentences and WordNet adjectives. We test relational biomedical opposites inside retrieval chains. BioHopR builds 1-hop and 2-hop questions from the PrimeKG biomedical knowledge graph. It reports that o3-mini's precision falls from 37.93% at 1-hop to 14.57% at 2-hop, and that medical open models are near zero at 2-hop. It evaluates closed-book generation and does not diagnose why hop 2 fails. We reuse its PrimeKG path construction and add the contrastive and retrieval layer.

The method draws on three sources. We reuse the query-side contrastive loss of the minimally-edited-questions paper and its released code (ytyz1307zzh/Minimally_Edited_Questions), and we move the contrast from the question to the post-hop state. IRCoT alternates chain-of-thought steps with retrieval. Q-RAG trains a small state/action embedder with reinforcement learning while the language model stays frozen (griver/Q-RAG). These two are the multi-step retrievers we probe and retrain. Neither has been tested with minimal-pair distractors.

Two papers constrain the design. Do Clinical Question Answering Systems Really Need Specialised Medical Fine Tuning? argues that general models with inference-time activation steering match or beat specialised ones on single-hop clinical true/false questions. This is why domain pre-training is treated as an experimental variable, not an assumed benefit. BioBART shows that domain pre-training on PubMed helps generation and that discourse order matters. It does not probe sensitivity to minimal perturbations, which is precisely the property we measure.

Research Program

The first component is the benchmark. We sample 2-hop paths from PrimeKG over drugs, diseases, proteins, and phenotypes, following BioHopR. For each path we create an edited twin. The twin flips the hop-1 relation to its domain opposite or swaps the hop-1 entity for a sibling of the same type. The edit is kept only if the new path exists in the graph and yields a different bridge entity and a different final answer set. Each edge is attached to a supporting passage. In the first tier, edges are verbalized by templates and then paraphrased by a language model. In a second tier, edges are linked to PubMed abstracts that state the relation, where such abstracts can be found.

The evidence for the original path is placed in the corpus as a high-overlap distractor for the twin, and the reverse holds as well. A random subset of pairs is checked by people with biomedical training. The minimally-edited-questions paper shows such checking is necessary for answers to be valid. We match token lengths across pairs, following the warning in Token-length Bias in Minimal-pair Paradigm Datasets. As a general-domain control, we build a smaller set of verified hop-1 edits on MuSiQue. MuSiQue's construction from single-hop sub-questions makes such hop-1 edits tractable.

The second component is the set of metrics. Pair consistency counts a pair as correct only if both questions are answered correctly. The pairwise scoring follows COM2SENSE. Hop-wise recall measures whether the gold passage is retrieved at each hop. Carryover rate is the fraction of edited questions whose hop-2 retrieval returns the original path's hop-2 evidence. Carryover rate is the direct signature of the failure we hypothesize.

The third component is the training fix. A multi-step retriever embeds a state $s_t$, consisting of the question plus the passages retrieved so far, and scores candidate passages $p$. Alongside the usual retrieval loss, we add a hinge term at each hop. Let $s_t$ be the state for the original question, $\tilde s_t$ the state for its edited twin, $p^+_t$ the gold passage for the original, and $\tilde p^+_t$ the gold passage for the twin. Writing $\mathrm{sim}$ for the retriever's similarity score and $m$ for a margin, the extra loss is

$$\mathcal{L}_{\text{hop}} = \sum_t \max\big(0,\; m - \mathrm{sim}(s_t, p^+_t) + \mathrm{sim}(s_t, \tilde p^+_t)\big).$$

This term is applied symmetrically for the twin. It forces the hop-2 state to depend on which hop-1 relation was asked, not merely on the shared words. We apply it to an iterative dense retriever, and to Q-RAG's embedder as a shaped reward term on chosen actions. In each case we compare a general-domain initialization with a PubMed-pretrained initialization of identical architecture.

Baselines

The naive reference point is an iterative dense retriever trained with the standard retrieval loss and in-batch negatives, with no contrastive signal. We run it on the new benchmark to establish carryover rates. The minimally-edited-questions paper reports that vanilla DPR loses 40–45% of passage-ranking MRR (mean reciprocal rank) on single-hop minimal edits. We expect a comparable or larger collapse at hop 2.

The strongest existing method for this exact property is that paper's query-side contrastive loss. It reports MRR rising from 0.515 to 0.565 on MEQ-GPT for DPR Large, and 11–12% Exact Match gains on MEQ-AmbigQA. We apply it to the first-hop question only, using the same encoder, the same training pairs, and the same number of gradient steps and hyperparameter search trials as our method. Beating it means a statistically significant reduction in hop-2 carryover rate and a gain in pair consistency, with standard-question recall not reduced. If it already brings hop-2 carryover near hop-1 levels, our first claim is false and we will report that.

For the language-model-driven side, the strongest competitor is IRCoT, which rewrites each hop's query in natural language. Its paper reports retrieval recall gains of 11–21 points over one-step retrieval across HotpotQA, 2WikiMultihopQA, MuSiQue, and IRC. The file gives no absolute MuSiQue score for IRCoT, and no reported result was found on Lacuna for Q-RAG's MuSiQue score. We therefore rerun both on our benchmark and on MuSiQue with the same corpus and the same retriever backbone. The general-domain reference on MuSiQue is the original paper's 47.3 F1 for Select+Answer, against 78.0 for humans.

We also include the generative pair selector of Generative Context Pair Selection for Multi-hop Question Answering as a reranker over the top candidates. It scores passage pairs by how likely they make the question and limited its accuracy drop on Adversarial HotpotQA to about 1%, against about 4% for a discriminative baseline. Its quadratic cost restricts it to reranking, so it serves as an upper-cost competitor rather than a first-stage retriever.

Two ablations isolate the key ingredient. The first replaces minimal-pair negatives with an equal number of hard negatives mined by BM25, a classic keyword-matching retriever. This tests whether minimality matters or merely hardness. The second applies minimal-pair negatives only at hop 1 versus at every hop.

Why not just…?

Why not just use an NLI-trained biomedical encoder? SemAntoNeg shows NLI training lifts negation handling to 93.6%, so this is a real threat for the relation-flip subset. The research file notes that NLI training has not been shown to fix entity swaps or relational role flips. It also does not address carryover, which arises from the retrieved context rather than the question. We include an NLI-tuned encoder as an initialization. If it matches our method on carryover for relation flips, we will narrow the claim to entity swaps.

Why not just apply the single-hop contrastive loss from the minimally-edited-questions paper? This is the main baseline described above. It is cheaper and may suffice. The experiment that settles it compares carryover at hop 2 with and without state-level negatives, holding everything else fixed.

Why not just let a strong language model rewrite each hop, as IRCoT does? Natural-language rewriting may resolve the bridge entity explicitly and avoid carryover. BioHopR's 14.57% 2-hop precision for o3-mini suggests that strong models still misresolve biomedical bridges. Agentic loops also cost one generation per hop. We report pair consistency per unit of inference compute. If IRCoT dominates at acceptable cost, the contribution reduces to the benchmark and the diagnosis.

Why not just rely on domain pre-training? Whether domain pre-training helps or hurts is the question itself, not an assumption. BioHopR's near-zero results for medical models and the steering results of the specialisation paper both cast doubt on it.

Decisive Checks

The work succeeds if three things hold. Hop-2 carryover substantially exceeds hop-1 error for baseline retrievers. State-level minimal-pair negatives reduce it more than both the question-level loss and BM25 hard negatives under matched budgets. The effect replicates on the MuSiQue control set.

Separately, the comparison between general and PubMed initializations should show whether domain pre-training raises or lowers carryover on co-occurrence-heavy relations, such as drug–indication. Either answer is informative to practitioners who default to domain encoders.

A null result on the first check, where hop-2 errors are no worse than hop-1 errors, would retire the propagation hypothesis. That would itself be worth reporting, because it would mean single-hop contrastive training is enough.