When a model answers "In what city was the director of Inception born?", a correct answer tells us little. The model may have read the evidence and followed the chain from film to director to birthplace. It may also have recalled "London" because the question, or the fact, appeared in its training data. Drug design had the same kind of problem. Some molecules score well against a target protein only because they stick to almost everything. The fix was a delta score: the molecule's binding score on the intended target minus its average score on unrelated targets. This proposal applies the same subtraction to multi-hop question answering. We score a system's answer on the real evidence chain, then subtract its average score for that same answer across freshly generated counterfactual chains. In those chains the intermediate "bridge" entity has been swapped for a comparable entity, so the evidence now implies a different answer. A system that reasons from the evidence changes its answer when the chain changes. A system that recites memorized answers gives the same answer everywhere, and its delta falls to zero.
The payoff is an evaluation that separates reasoning from recall on real, natural-language entities, which existing contamination-resistant tests do not do. It also resists contamination: counterfactual worlds are generated on demand and their answer keys are never published. The same probe can then test a claim that is widely assumed but not established, namely that making a classifier write out its rationale before answering causes it to follow the evidence rather than its priors.
Core Claim
We claim three things. Raw multi-hop F1 over-credits systems in the same way raw docking scores over-credit "sticky" molecules. Evidence-conditioned delta scoring, with a carefully designed background set of counterfactual chains, will reorder systems. That reordering will track genuine reasoning, measured on a fully fictional reference world, better than F1 or existing shortcut probes do.
Formally, for a question $q$ with original evidence $E$ and original answer $a$, let $s(a \mid E)$ be the system's score for $a$. The score is either the answer probability or 0/1 exact match. Let $E_1,\dots,E_N$ be counterfactual evidence sets in which the bridge entity is replaced. The reasoning delta is
$$\Delta(q) = s(a \mid E) - \frac{1}{N}\sum_{j=1}^{N} s(a \mid E_j).$$
This mirrors the definition in Delta Score: Improving the Binding Assessment of Structure-Based Drug Design Methods. That paper subtracts mean off-target docking affinity from on-target affinity. Doing so reversed the apparent ranking: ground-truth ligands scored −0.810, the Targetdiff generator −0.382, and random ZINC molecules −0.019, even though random molecules beat ground truth on 22.4% of targets under raw Glide scores. We report two companion quantities next to $\Delta$. The follow rate is how often the system outputs the answer implied by each counterfactual world. The persistence rate is how often it outputs the original answer despite evidence that contradicts it.
How the Counterfactual Worlds Are Built
The construction reuses the compositional structure of existing benchmarks. MuSiQue composes each multi-hop question from single-hop sub-questions joined by an explicit bridge entity. 2WikiMultihopQA uses Wikidata relations with annotated supporting facts. Both make it easy to locate the bridge entity $b$ and the relation that carries it to the answer.
For a two-hop question we sample a replacement $b'$ of the same type, such as another film director. We then minimally edit only the hop-1 supporting paragraph so that it names $b'$ instead of $b$. The hop-2 paragraph is replaced by the real Wikipedia or Wikidata fact about $b'$, which gives answer $a'$. The only false statement in the world is therefore the edited hop-1 sentence, and the correct evidence-conditioned answer $a'$ is itself a true fact. The only way to reach $a'$ is to follow the chain.
A mirror variant edits hop 2 instead. Comparing the two variants localizes which hop a system ignores. For three- and four-hop MuSiQue questions, the swap position is sampled, and an LLM rewrites edited sentences under human verification of a sample. Outputs that flag the evidence as conflicting are counted separately rather than scored as errors.
The choice of background is the open problem that Direction 1 identifies. As in the drug-design work, it determines what the delta means. Following CaseOLAP's split of entity relevance into popularity and distinctiveness (Data-Driven Insights into the Association Between Oxidative Stress and Calcium-Regulating Proteins in Cardiovascular Disease), we draw replacements $b'$ from three pools:
- Popularity-matched: same type, same frequency band as $b$.
- Random same-type: any entity of the right type.
- Most-popular: the most famous entities of that type.
Popularity is measured by entity frequency in pretraining text, in the style of Snoopy. That work showed few-shot accuracy rising with term frequency in The Pile. Popularity-matched swaps are the primary background, because they hold the model's prior familiarity roughly constant between the real and counterfactual worlds.
Research Program
The first study is a measurement study. We compute $\Delta$, follow rate and persistence rate on the dev/test splits of MuSiQue-Ans, 2WikiMultihopQA and HotpotQA. The systems are closed-book models, one-shot dense retrieval, IRCoT and HippoRAG, each on a small open model (Flan-T5-XL, as in IRCoT), a 7–13B instruction-tuned open model, and one API model.
We test four predictions. First, $\Delta$ falls as the bridge entity's pretraining frequency rises, which is the popularity bias seen in Snoopy and CaseOLAP. Second, the systems ranked by $\Delta$ differ from their ranking by F1, the text analogue of the ZINC paradox. Third, persistence rises with hop depth. DyVal found that reasoning depth hurts faster than width on its synthetic reasoning tasks, and this prediction asks whether the same holds on natural text. Fourth, retrieval systems score higher on $\Delta$ than closed-book models, but not in proportion to their F1 gains.
The second study concerns explanation-augmented classification, meaning models that write a rationale before giving the label or answer. Work in that direction, such as CAGE and joint rationale losses, assumes that writing the rationale forces the model to use the evidence. The counterfactual worlds let us check this directly. We measure whether the rationale names the swapped bridge $b'$, whether the final answer is $a'$, and how often the two disagree. The most telling case is a rationale that correctly cites $b'$ while the answer stays $a$, which is post-hoc rationalization caught in the act. DyVal's observation that 33.8% of GPT-4's failures were answers with no intermediate reasoning is consistent with this failure mode, but it does not measure it on natural-language evidence.
The third study is an intervention. We fine-tune the open 7–13B model on counterfactual worlds generated from the training splits. Four conditions share the same data volume and compute:
- rationale plus answer
- answer only
- rationale plus answer on original worlds only
- no fine-tuning
These conditions separate two effects: supervising the rationale, and exposing the model to evidence that conflicts with its memory.
The last component is contamination. Test worlds are regenerated with fresh seeds for each evaluation, and their answer keys stay private. A search-augmented agent that retrieves a leaked MuSiQue answer key will output $a$ in counterfactual worlds, which lowers its $\Delta$. The metric therefore also acts as a detector for the leakage documented in Search-Time Data Contamination, where Sonar Deep Research rose from 17.4% to 37.5% on contaminated HLE questions. We run an open search-augmented agent to check this detection effect.
Relation to Prior Work
Three papers motivate the gap by showing that multi-hop scores overstate reasoning.
- Is Multihop QA in DiRe Condition? removes supporting facts. It found that only 17.5 of XLNet's 72 HotpotQA F1 points required connecting facts.
- MuSiQue filters out questions answerable without the first hop.
- Counterfactual Multihop QA subtracts learned shortcut effects at inference time. It lowered BERT's disconnected-reasoning score from 64.1 to 54.6.
All three detect models that answer without connecting facts. None detects models that ignore connected facts in favour of memorized ones. Removal-based probes cannot detect this, because recall still works when a fact is deleted. Our swaps put the evidence and the memory in conflict, which exposes that failure.
Two papers constrain the design.
- INVITE builds fresh test items by swapping entities in DBpedia triples, but only for single-hop invalid questions. Its authors name multi-hop extension as future work. We extend the swapping idea to valid multi-hop chains with changed answers.
- DyVal generates contamination-proof items, but only for formal math and logic tasks. We borrow its fresh-per-evaluation generation for natural text.
HippoRAG's node specificity, an inverse-document-frequency weight on graph nodes, is the nearest use of "penalize ubiquity" in multi-hop work. It adjusts retrieval. We use the principle for evaluation, and HippoRAG serves only as one of the systems we test.
Baselines
The naive reference is standard exact-match and F1 on the unmodified test sets, the number every leaderboard reports. For MuSiQue the file gives a best reported model F1 of 47.3 against 78.0 for humans (MuSiQue). F1 must be reported, but it cannot separate reasoning from recall, so it sets the floor that any diagnostic must improve on.
The simplest competing diagnostic is context gain: open-book accuracy minus closed-book accuracy for the same model on the same questions. It is also a subtraction, and it is cheap. We compute it on the identical question set with the identical models and prompts.
The strongest existing diagnostic is the disconnected-reasoning (DiRe) probe and its sufficiency transform (CSST) from Trivedi et al. We rerun it with their published fact-subset procedure on HotpotQA and on the MuSiQue questions we use. The reported reference point is 17.5 genuine-reasoning F1 out of 72 for XLNet on HotpotQA. It is the strongest competitor because it already decomposes a score into reasoning and non-reasoning parts.
For the intervention study, the strongest training baseline is Counterfactual Multihop QA. It is reported to cut BERT's disconnected-reasoning F1 from 64.1 to 54.6 while raising the true-reasoning score by 5.8 points. We reimplement it on the same backbone and training data as our fine-tuned model, with the same number of hyperparameter trials.
No reported result on Lacuna was found for any evidence-swap delta metric on these benchmarks. The ablations are therefore internal:
- an answer-only model trained on counterfactual worlds, which isolates the rationale ingredient
- a rationale model trained on original data only, which isolates the conflicting-evidence ingredient
- the three background pools, which isolate background design
The success criterion for the diagnostic comes from the fictional reference world described below. $\Delta$ beats a competitor if its rank correlation with fictional-world accuracy across all systems is higher, with a bootstrap 95% interval that excludes zero difference. For the intervention, the rationale-trained model must raise follow rate on held-out swap worlds over answer-only training by more than the bootstrap interval, without losing more than two F1 points on the originals.
Why not just…?
Why not just subtract closed-book accuracy from open-book accuracy? Context gain measures whether retrieval adds information. It does not measure whether a model defers to evidence when evidence and memory disagree. A model that knows the answer closed-book gets zero context gain even if it would faithfully follow any evidence, so the diagnostic penalizes knowledge rather than detecting recall. Context gain is also a single number per question and cannot say which hop was ignored. It is nevertheless a real threat, and it is the first competitor in the validity test. If its correlation with fictional-world accuracy matches that of $\Delta$, we will report that the cheap metric suffices.
Why not just evaluate on fully fictional entities, as DyVal does for formal tasks? Fictional worlds remove the model's priors entirely, so they cannot measure how priors interfere with evidence on real entities. That interference is what matters for domain use and what the popularity-stratified analysis targets. We use fictional worlds as the ground truth for validating $\Delta$, not as its replacement. They are built by renaming every entity in a sample of 2WikiMultihopQA chains to invented names. If the fictional score alone ranks systems exactly as $\Delta$ does and shows no popularity effect, the real-entity machinery adds little, and the experiment will reveal this.
Why not just use the DiRe probe or CSST? Those methods delete facts, and deletion cannot distinguish a model that joins the remaining facts from one that recalls the answer, since recall survives deletion. The swap design separates the two by making recall produce a wrong answer. The direct test runs both probes on the same systems and compares their correlation with fictional-world accuracy.
Why not just add contrastive delta scoring to multi-hop retrieval instead? This is the obvious combination of the four directions. HippoRAG's node specificity already down-weights hub entities during graph search, so an external-background version would probably be a marginal variant of it. The evaluation question is unanswered, so we put effort there.
Why not just audit retrieved URLs to handle contamination? Auditing catches leakage only at search time and only from known mirrors. It does nothing about answers memorized during pretraining. Swap worlds catch both kinds, because either produces persistence on $a$. We confirm this by planting a leaked answer key in the search index and checking that $\Delta$ drops.
Decisive Checks
The central test is validity. Across the full grid of systems we compute Spearman correlation between each diagnostic on real-entity benchmarks and accuracy on the fictional reference. The diagnostics are F1, context gain, DiRe genuine-reasoning score and $\Delta$ under each background pool. The proposal succeeds if popularity-matched $\Delta$ correlates best and the popularity-stratified decline in $\Delta$ appears. It fails, and should be reported as failing, if context gain does as well.
The second test concerns rationales. We measure the rate of rationale–answer disagreement on swap worlds, and whether rationale-supervised training on conflicting evidence raises follow rate beyond answer-only training. The third test is sensitivity to background design. If rankings flip between popularity-matched and most-popular backgrounds, background choice is as consequential in text as in drug design, and it has to be standardized before delta scores are compared across papers.
References
- Delta Score: Improving the Binding Assessment of Structure-Based Drug Design Methods
- ♫ MuSiQue: Multihop Questions via Single-hop Question Composition
- Data-Driven Insights into the Association Between Oxidative Stress and Calcium-Regulating Proteins in Cardiovascular Disease
- Snoopy: An Online Interface for Exploring the Effect of Pretraining Term Frequencies on Few-Shot LM Performance
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
- HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Search-Time Data Contamination
- Is Multihop QA in DiRe Condition? Measuring and Reducing Disconnected Reasoning
- Counterfactual Multihop QA: A Cause-Effect Approach for Reducing Disconnected Reasoning
- INVITE: a Testbed of Automatically Generated Invalid Questions to Evaluate Large Language Models for Hallucinations