Breaking One Hop: Diagnostic Distractors for Multi-Hop Reading Questions

Multiple-choice questions are only as informative as their wrong answers, which are called distractors. A good distractor attracts readers who have a specific gap in understanding and is unambiguously wrong for readers who don't. For questions that require chaining facts across documents ("In what city was the director of Inception born?"), there is a natural way to build such wrong answers. Solve the question step by step, then corrupt exactly one step. Readers can stop after the first fact and answer "Christopher Nolan". They can follow a plausible but wrong bridge entity to another director's birthplace. Or they can grab a city that merely appears near the question words. Each corruption produces an option that is attractive only to a reader whose reasoning broke at that step.

We propose to generate multi-hop multiple-choice items in which every distractor is indexed to a particular hop and a particular failure type. We then test a specific claim: which hop is broken, and how semantically far that hop's evidence lies from the question, predict how often real examinees choose the distractor and how hard the item is. If this holds, item difficulty can be set by deciding which step to corrupt rather than by asking a language model for "a harder question." A student's wrong answer would also say where their reasoning chain failed.

Core Claim

Existing distractor generators work on single sentences or single paragraphs, and existing multi-hop difficulty work produces question–answer pairs without distractors. On the distractor side, DisGeM masks an answer span, decodes replacements, and filters them with sentence-level entailment checks. The Pseudo-KL regulated generator constrains word-level cloze distractors. On the multi-hop side, GRADE and MHTS build multi-hop question–answer pairs whose machine error rates track hop count and query–evidence distance, but they produce no options. The closest prior idea is the "Hard Chain-of-Thought" approach summarized in The Mechanics of Automated Distractor Generation. It solves an item step by step and then perturbs links to create near-miss options. Two things are missing from it: a notion of which link to break, and any measurement of what breaking it does. The file reports no numbers for it.

Our claim is narrower and testable. In multi-hop text questions, the attraction rate of a distractor is governed mostly by two properties of the hop it corrupts: its position in the chain and its evidence distance. Evidence distance is GRADE's measure of how dissimilar the hop's supporting passage is to the question. Together these give a continuous difficulty control that can be calibrated with item response theory, the standard psychometric model that places items and examinees on a shared ability scale. The setting is multi-hop reading. Chain-of-thought prompting, where a model writes intermediate reasoning steps before answering, is the means of exposing the steps to corrupt. Proficiency awareness is the evaluation target.

Research Program

Generation pipeline. The generator first obtains an explicit reasoning chain for each source question. On MuSiQue this comes from the gold single-hop decompositions. On HotpotQA, which provides supporting facts and bridge entities but no sub-questions, it comes from an interleaved retrieve-and-reason solver in the style of IRCoT. IRCoT alternates chain-of-thought sentences with retrieval and was shown to raise multi-hop retrieval recall by 11–21 points.

For each hop $k$, the generator applies three typed corruptions:

  • Early stop. The intermediate answer at hop $k$ becomes the option.
  • Wrong bridge. A sibling entity is taken from a non-gold passage and the remaining hops are re-executed from it, so the option is a correctly computed answer to a slightly wrong chain.
  • Lexical shortcut. The option is a type-matched span chosen for its surface overlap with the question.

Each candidate carries a tag recording its hop position $k$, its failure type, and the evidence distance $d_k$ of the corrupted hop.

Double-key check. Sentence-level entailment checks like DisGeM's cannot tell whether a multi-hop alternative is secretly correct, because validity depends on facts spread across documents. We therefore verify each candidate against the full gold chain. A retrieval-augmented solver is asked to justify the candidate as the answer, and an entailment model checks each step of that justification against its passage. A candidate survives only if no step-supported justification is found. Humans then adjudicate a sample to estimate the residual rate of options that are defensibly correct.

Psychometric calibration. Items are administered to two examinee populations. The first is a panel of language-model examinees spanning capability: small to large models, closed-book versus retrieval-augmented, and with versus without chain-of-thought. This follows the proxy-examinee approach of Uto et al.. The second is human readers, because Brains vs. Algorithms? found that an expert-favored AI distractor was chosen by 0% of students, so machine or expert plausibility cannot stand in for human attraction.

We fit a nominal-response item response model. This variant of item response theory models the probability of choosing each specific option, not just correct versus incorrect. We then regress the fitted option-attraction parameters on the hop tags:

$$a_{j} = \beta_0 + \beta_1 k_j + \beta_2 d_{k_j} + \gamma_{\text{type}(j)} + \epsilon_j$$

In this equation:

  • $a_j$ is the fitted attraction of distractor $j$.
  • $k_j$ is the position of its corrupted hop.
  • $d_{k_j}$ is that hop's evidence distance.
  • $\gamma_{\text{type}(j)}$ is an offset for each failure type.
  • $\epsilon_j$ is residual noise.

If the claim holds, these few terms explain most of the variance in attraction. The fitted model is then inverted to choose corruptions that hit a target item difficulty.

Diagnostic validity. For a subset of human examinees, we also administer the single-hop sub-questions separately. This lets us test whether choosing the hop-$k$ distractor predicts failing sub-question $k$ specifically.

Baselines

Naive reference point. Distractors are drawn as random type-matched entities from the HotpotQA distractor paragraphs. This shows how much plausibility comes for free from same-type entities in context.

Strongest existing method. A frontier model (GPT-4o) is given the passages, question, and answer. It is prompted with chain-of-thought and few-shot examples retrieved by question similarity, following predictive prompting, to write three plausible distractors at a stated difficulty level. This is the strongest competitor because it is what practitioners would actually do, and because frontier models can plausibly infer bridge entities themselves.

Fairness rules apply to this baseline. It receives the same passages, the same number of candidate generations per item, the same double-key verifier, and the same prompt-tuning budget we spend on our own prompts. On the generation side, the file's relevant reported numbers are a 42.6% match to targeted inference types for GPT-4o in Ma et al., and 27.8% student-distractor prediction accuracy for chain-of-thought prompting of Claude-3.5-Sonnet in the MCTS reconstruction paper. No reported result was found on Lacuna for any method's distractor attraction or item-response-calibrated difficulty on multi-hop reading items. The baselines will therefore be run fresh under identical conditions.

Graph-walk baseline. A walk over an entity graph built from the corpus that avoids the gold answer's neighbors, adapting KGGDG. In the medical setting KGGDG lowered LLM accuracy by about 10 points, but it maximizes difficulty rather than calibrating it.

Unindexed-perturbation ablation. This is Hard-CoT-style corruption of a random step without hop tags, and it isolates our key ingredient. A second ablation removes the multi-premise verifier.

What counts as beating the baselines. We claim success if three things hold:

  1. The hop-tag regression predicts held-out item difficulty with substantially higher rank correlation than the stated-difficulty labels of the GPT-4o baseline.
  2. Our distractors match or exceed the baseline's mean human attraction at an equal or lower human-adjudicated double-key rate.
  3. Hop-indexed distractors show diagnostic specificity that the baselines lack, measured as a higher probability of failing sub-question $k$ given selection of the hop-$k$ distractor.

Why not just…?

Why not just prompt GPT-4o to write harder or easier distractors? This is the real threat, and it is the main baseline. Ma et al. found that chain-of-thought gave no significant gain in hitting targeted inference types and reached only 24.1% on text-connecting inferences. That suggests stated difficulty labels are weakly controllable, but it was measured on ten passages and without distractors. Our calibration experiment settles the question directly. If the baseline's stated difficulty correlates with fitted item difficulty as well as our hop tags do, the structured control adds nothing, and we will report that.

Why not just use MuSiQue's gold intermediate answers as distractors, with no chain-of-thought at all? On MuSiQue this is essentially our early-stop corruption with oracle steps, so we treat it as the upper bound for that failure type rather than a competitor. The open question is whether chain-of-thought-recovered chains on HotpotQA, which has no gold decompositions, approach the oracle. Running the same pipeline with gold versus recovered chains measures that gap. If the gap is large, the contribution narrows to the hop-indexed taxonomy and its psychometrics, which still stands.

Why not just maximize semantic similarity to the key, or walk a knowledge graph away from it? Similarity-driven options risk being defensibly correct. That is the failure DisGeM's entailment filter addresses only within a single sentence. Graph walks such as KGGDG's push difficulty up but provide no dial. The Brains vs. Algorithms result warns that maximal theoretical plausibility can mean zero human uptake. Our human study directly compares the attraction distributions and double-key rates of these baselines with ours.

Proposed Experiments and Decisive Checks

Item pool. We build roughly 600 items: 2–4-hop questions from MuSiQue-Ans, with gold decompositions, plus bridge questions from HotpotQA in the distractor setting, without gold decompositions. Each item has four options produced by each method.

Examinees. The language-model panel consists of about twenty configurations. The human study recruits adult crowdworkers who read the gold and distractor passages. Each worker answers a balanced subset, so that every item collects enough responses to fit the nominal-response model. Sub-question probes go to a subset of workers.

Decisive check 1: difficulty control. On held-out items, we compute the Spearman correlation between target and fitted difficulty, and between predicted and observed distractor attraction. We also check that harder targets yield monotonically lower human accuracy.

Decisive check 2: diagnostic specificity. We measure the lift in sub-question-$k$ failure among examinees who chose the hop-$k$ distractor, compared with those who chose other distractors.

Decisive check 3: shortcut resistance. This borrows the logic of ReClor, where options-only models reached 78.9% on its biased split. A model trained on options alone, and a single-paragraph reader given only one gold passage, should both score near chance on our items. If they do not, distractors are being rejected by surface cues and the multi-hop claim fails.

Decisive check 4: machine–human agreement. We correlate the language-model panel's item parameters with the human parameters. This tells us whether the proxy examinees used by Uto et al. can be trusted for multi-hop items, or whether, as Brains vs. Algorithms suggests, calibration must stay human.

What would falsify the proposal. The central claim fails if hop position and evidence distance explain little of the human attraction variance, or if the GPT-4o baseline matches our difficulty correlation. In either case we would report the negative result together with the released item bank and response data.