Near-Miss Distractors: Stress-Testing Multi-Hop Reasoning with Grammar-Preserving Substitutions

Multi-hop question answering chains facts together. To answer "In what city was the director of Inception born?", a system first finds the director and then looks up that person's birthplace. Current systems are usually tested for robustness by mixing in irrelevant paragraphs, which are text about other topics. The hardest distractor in real corpora is a different kind of text. It is a sentence that looks almost exactly like the evidence the system needs, with the same words, the same entities and the same grammar, but with the roles swapped or one entity replaced, so that it states a different fact. We propose to build such "near-miss" distractors automatically. The method is part-of-speech constrained substitution: words in a gold evidence sentence are replaced or swapped only with words of the same grammatical category (noun for noun, verb for verb), so the result stays fluent and grammatical. We then insert these distractors at a chosen hop of a reasoning chain and measure how errors at that hop propagate to the final answer.

The motivation comes from a result outside question answering. PAWS used grammar-preserving word swaps to build sentence pairs that share almost all their words but differ in meaning. BERT, which scores above 90% on Quora Question Pairs, fell to 33–40% on these pairs. Adding 12,000 such examples to training restored 85%. Multi-hop systems choose their next retrieval query and their bridge entity largely from lexical overlap with the current evidence. That makes them a natural target for the same failure, and a cascading one: a near-miss accepted at hop one corrupts every later hop. No existing multi-hop noise study tests this. We propose a controlled instrument that does, and we test whether the PAWS-style fix, training on the near-misses, carries over to reasoning chains.

Core Claim

We claim three things. First, multi-hop pipelines fail on grammatical, high-overlap distractors that swap relational roles, even when those distractors are few, while standard topical distractors hurt them much less. Second, the damage depends on where the distractor sits. A near-miss at an early hop should hurt more than one at the final hop, because a wrong bridge entity poisons all later retrieval. Third, the part-of-speech constraint matters for its own sake. It holds grammaticality and lexical overlap fixed, so a failure can be pinned on relational binding (who did what to whom) rather than on detectable artifacts. The same substitutions without the constraint, which yield ungrammatical text, should be easier for models to ignore.

The contribution is a measurement tool plus one targeted fix. It is not a new retrieval architecture.

Relation to Prior Work

Two strands of work motivate this proposal.

The first strand supplies the substitution method. PAWS showed that models rely on bag-of-words overlap. Lymperopoulos et al. framed minimal part-of-speech-matched edits as an assignment problem and flipped classifiers while changing about 1% of tokens. Maheshwary et al. used part-of-speech filtering to keep black-box attacks grammatical. All three target single-sentence classifiers. None asks what a grammatical near-miss does inside a reasoning chain.

The second strand motivates the question and supplies the systems under test. Reason-KE teaches models to acknowledge, judge and selectively apply facts. It reports 90.2% multi-hop accuracy on MQuAKE-CF and only a 6.3% drop with four distractor facts. Those distractors, however, are unrelated declarative facts rather than rewritten versions of the needed fact. HANRAG discards documents with an LLM relevance discriminator. Superposition Prompting prunes whole documents by saliency. CompAct compresses retrieved text iteratively. Cho et al. show that zero-shot readers are distracted by irrelevant retrieved documents and repair this with an "unanswerable" escape and query-likelihood rescoring. Every one of these methods was tuned against topically irrelevant noise. Our instrument asks whether their gains survive noise designed to look relevant.

ChainRAG is the closest work on hop-level failure. It shows second-hop Recall@2 dropping by up to 18% when sub-questions use vague references, and it fixes this by rewriting. We borrow its focus on the individual hop, but we study adversarial evidence rather than underspecified queries.

MuSiQue constrains the benchmark choice. It showed that single-paragraph models score 64.8 F1 on HotpotQA but only 32.0 on MuSiQue. Our main experiments therefore use MuSiQue's connected questions, so that a model cannot ignore the corrupted hop and still answer correctly. DRAG is the one paper in the file that joins lexical variation and noise. It handles synonymy in queries and calibrates high-risk tokens, but it applies no grammatical constraint and does not examine hop-wise cascades.

Research Program

Constructing distractors. For each MuSiQue-Ans question we use the gold decomposition and intermediate answers to locate the supporting sentence for each hop. We tag each sentence with a part-of-speech tagger and generate three families of near-misses.

  • Role swaps exchange two same-category noun phrases, in the style of PAWS. "A directed B" becomes "B directed A."
  • Entity substitutions replace the bridge entity or the answer with a same-category, same-type entity drawn from the question's other paragraphs.
  • Relation substitutions replace a verb or adjective with a same-category embedding neighbour, for example "born" becoming "died." Candidates are ranked by language-model fluency.

Each distractor then goes through three checks:

  1. It must not be entailed by any gold paragraph, checked with an entailment model and a spot-check.
  2. It must contradict or diverge from the gold fact.
  3. A sample must pass human ratings of naturalness and of detectability as machine-edited.

Tagging errors are a known weakness of this pipeline, so we log the tagger's confidence and report results with low-confidence edits excluded as a robustness check. Each distractor is planted inside an otherwise irrelevant paragraph that sits in the retrieval pool or the fixed context. The controlled variables are the hop position, the number of distractors, and the lexical overlap with that hop's sub-question.

Measurement. MuSiQue provides intermediate answers, so we score more than final F1 and EM.

  • Hop accuracy: whether the system's bridge entity at each hop matches the gold one. For systems that expose chains, this is read from the chain or from logged retrieval queries.
  • Adoption rate: how often the output contains the entity planted by the distractor.
  • Cascade ratio: how much more likely the final answer is to be wrong given a wrong bridge at hop $k$, compared with wrong-bridge rates in clean runs.

The same construction transfers to MQuAKE-CF, so Reason-KE can be tested on its home benchmark with near-miss distractors in place of its original ones. It also transfers to HotpotQA's distractor setting.

Mitigation. We test whether a PAWS-style fix carries over to reasoning chains. We fine-tune a small relevance filter, which scores each candidate passage against the current sub-question and bridge entity, on near-miss negatives generated from MuSiQue training questions. We then place this filter in front of IRCoT and HANRAG. For Reason-KE we instead add near-miss facts to its training trajectories. All distractor generation for training uses only training-split questions.

Baselines

Naive one-shot retrieval. This is a single retrieval step followed by reading, and it is the reference point. Its decline under topical versus near-miss distractors calibrates the effect size.

Topically irrelevant distractors. The same systems receive distractors matched in count and length but drawn from unrelated paragraphs. This is the standard noise condition in HotpotQA and in Reason-KE. The claim fails if near-miss distractors do not hurt reliably more than these.

Strongest existing multi-hop systems. On MuSiQue these are IRCoT, which alternates chain-of-thought sentences with retrieval, and HANRAG with its relevance discriminator. Both are the strongest noise-aware multi-hop pipelines in the file. For reference, MuSiQue's own best model reached 47.3 F1 against 78.0 for humans (MuSiQue). HANRAG reports only relative gains over Adaptive-RAG, nearly 7% EM, so no absolute MuSiQue figure from it was found on Lacuna. We therefore rerun it ourselves on clean data first.

Reason-KE on MQuAKE-CF. This is the strongest distractor-resilient method, at 90.2% accuracy with a 6.3% drop under four distractors (Reason-KE). We first reproduce its clean and original-distractor numbers, then replace its distractors with near-misses at the same count. Beating it here does not mean higher accuracy. It means showing that its reported drop grows substantially, for example to double digits, under near-misses. The mitigation then counts as a success if it brings the drop back close to 6.3% without lowering clean accuracy.

Fairness. Every system uses the same retriever, corpus, backbone LLM where swappable, and context budget. The same distractor sets are injected with fixed seeds. The mitigation filter gets the same fine-tuning budget as a control filter trained on topical negatives.

Ablations isolating the constraint. Two ablations test the part-of-speech constraint directly. The first applies identical substitutions without the category filter, producing ungrammatical near-misses. The second uses LLM-written counterfactual sentences matched in lexical overlap to our distractors.

Why not just…?

Why not just let an LLM write the counterfactual distractors? This is a real threat to the claim that the part-of-speech constraint is special. LLM-written contradictions may be just as effective. Their weakness is that they vary wording, length and overlap together, so it cannot be said which property caused a failure. The matched-overlap LLM condition in our ablations settles the question. If it is equally harmful, we report the phenomenon as "grammatical high-overlap near-misses." The constrained generator would then be the cheaper, controllable way to produce them, not the only one. We would not claim a unique role for part-of-speech tags.

Why not just use HotpotQA's distractor setting or MuSiQue-Full's unanswerable questions? Those distractors come from retrieval or composition and share topic, not relational structure, with the gold facts. MuSiQue-Full tests whether a model can notice that evidence is missing, not whether it can refuse evidence that is wrong but plausible. The topical-distractor baseline measures directly how much harder near-misses are. If the gap is small, existing benchmarks suffice and our instrument adds little.

Why not just rely on explicit relevance reasoning like Reason-KE or HANRAG's discriminator? These methods may well already resist near-misses. That would itself be a useful finding, and our tests are designed to show it. Our prior points the other way. Their relevance judgments ask whether a fact is about the query, and a role-swapped fact is about the query. PAWS showed strong encoders failing exactly at this kind of distinction until they were trained on it.

Why not just scale the backbone model? We include two backbone sizes for each system. If near-miss sensitivity largely disappears with scale, the mitigation matters less, and we will say so.

Proposed Experiments and Decisive Checks

The first decisive check is a sweep over hop position, from the first hop to the last, on MuSiQue 2- to 4-hop questions. It crosses three distractor types (near-miss, topical, unconstrained ungrammatical) with one, two and four distractors per question. The central prediction is a clear gap between near-miss and topical distractors, with the gap largest at early hops. If the adoption rate and cascade ratio show no hop dependence, the cascade claim is wrong.

The second check is the grammaticality ablation. If ungrammatical substitutions are as harmful as grammatical ones, the constraint is not what drives the effect, and we drop that claim.

The third check is transfer. The near-miss-trained filter must recover a substantial share of the near-miss loss for IRCoT and HANRAG, more than the topical-negatives control filter does, while keeping clean MuSiQue F1 within noise. We also test it on HotpotQA and on MQuAKE-CF distractors that it never saw during training, to check that it has not simply learned the generator's artifacts. Human detectability ratings on a sample of the distractors confirm that failures reflect genuine confusion rather than obvious editing marks.