Multi-hop question answering benchmarks such as HotpotQA, 2WikiMultiHopQA and MuSiQue are built from Wikipedia, which sits inside the pretraining data of every model they evaluate. A model that "answers" a three-step question may never have read the retrieved passages; it may simply remember the answer. Counterfactual rewriting addresses this by changing the facts in the passages so that memory and context disagree. That is the main remedy available today, and it does not scale, because someone has to check that each rewritten passage still says exactly what was intended. We propose to automate that check. Each supporting passage is broken into atomic propositions, meaning the smallest standalone factual statements it contains. Individual propositions are edited, and the passage is rewritten from the edited set. Then a different model extracts propositions again from the rewritten text, and the passage is accepted only if the re-extracted set matches the intended set. This is the "round-trip" or back-translation check used to verify formal translations, applied here to text generation instead.
Because edits happen one proposition at a time, the benchmark can also do something whole-question counterfactuals cannot. It can make some hops of a question counterfactual and leave others factual, then read off from the model's answer which individual hops it took from memory rather than from the provided evidence. The deliverable is a contamination-resistant multi-hop benchmark that can be regenerated on demand, together with a per-hop map of where retrieval-augmented systems quietly fall back on what they memorized in pretraining.
Core Claim
The closest existing work is CofCA, which rewrites Wikipedia entities into counterfactual alternatives and has humans verify the results. It shows the problem is real: GPT-4 exact match falls from roughly 70% on factual data to roughly 53% on 2-hop counterfactual data, and only 36.3% of its intermediate reasoning chains are correct. CofCA is capped at 1,800 samples by human review, and it edits whole questions. We claim three things. First, round-trip verification at the level of atomic propositions can replace most of that human review with a measured error rate. Second, it can do so at the scale of MuSiQue's 25K questions and be regenerated with fresh counterfactuals at any time. Third, toggling counterfactuality per hop turns contamination from a yes/no property of a benchmark into a per-hop measurement of a model.
The intersection of the four directions is not empty, and each contributes something specific. BRIEF already decomposes documents into atomic propositions and composes synthetic multi-hop questions through bridge entities, but it uses this to train a compressor and evaluates on static Wikipedia benchmarks. DyVal generates fresh multi-step problems from graphs, with controllable depth, but only in arithmetic, logic and algorithmic domains rather than open-domain text. Faithful Autoformalization via Roundtrip Verification and Repair supplies the method we reuse. It translates statutory rules to logic, translates them back to text, re-formalizes them, and proves equivalence with an automated solver. It also warns about the main risk, "canceling errors," where the forward and re-formalization steps make the same mistake and a false pass results. Our proposal differs from all three because it uses the round trip to certify generated evaluation text rather than model outputs, and it uses atomic units as the unit of counterfactual editing rather than of retrieval.
Research Program
We start from MuSiQue (MuSiQue). Its questions are composed bottom-up from single-hop sub-questions with known bridge entities and supporting paragraphs, so the hop structure is given rather than inferred. MuSiQue is also already filtered against single-paragraph shortcuts.
Decomposition. For each supporting paragraph, an extractor model produces atomic propositions. We then align each sub-question to the proposition that answers it, which we call the hop proposition.
Editing. To make hop $k$ counterfactual, we replace the object of its hop proposition with a type-matched, plausible alternative, such as a different city or a fictional person with a consistent birth year. We then propagate the change. The next hop's paragraph must now be about the new bridge entity, so its subject propositions are rewritten as well. A small symbolic check confirms that chaining the edited hop propositions as subject–relation–object steps reaches a counterfactual final answer distinct from the factual one.
Rewriting and round-trip verification. A rewriter turns each edited proposition set back into a fluent paragraph. A re-extractor from a different model family, chosen to reduce canceling errors, then decomposes the rewritten paragraph. The two proposition sets are matched with a bidirectional entailment model (natural language inference). A paragraph is accepted only if all three of the following hold:
- every intended proposition is entailed by some re-extracted proposition;
- no re-extracted proposition contradicts the intended set;
- the original, pre-edit hop proposition is not entailed.
The third test is a leakage check that catches rewrites which quietly keep the true fact alongside the new one.
Repair. Failures go to scoped repair, following the autoformalization work. A diagnoser decides whether the rewrite or the extraction leg failed, and only that leg is regenerated. Because that paper found a strong "first-stage bias" in one diagnoser model, we log diagnosis accuracy against human labels rather than trusting the diagnoser blindly.
Mixed-hop toggling. For an $h$-hop question, we generate all $2^h$ patterns of factual and counterfactual hops. Each pattern changes which entity the chain passes through, so we design them so that every pattern yields a distinct final answer. A model's final answer then identifies which path it actually followed. If the context makes hop $k$ counterfactual but the answer matches the path that keeps hop $k$ factual, the model used memory at hop $k$. We define the per-hop memory-override rate $M_k$ as the fraction of questions where hop $k$ is counterfactual in context but the model's answer corresponds to the factual value at hop $k$. We also elicit intermediate answers and compare the two signals. A closed-book run, with the question and no context, gives each model's parametric baseline for each hop proposition, so we can test whether override concentrates on facts the model already knows.
Baselines
The naive reference point is the original static benchmark, factual MuSiQue paragraphs with no editing. Its reported best model reaches 47.3 F1 against 78.0 F1 for humans (MuSiQue). We run every evaluated system on factual MuSiQue and on our counterfactual version with identical retrievers, prompts and decoding. Any accuracy gap between the two is the contamination effect the static benchmark hides.
The strongest existing method for building counterfactual multi-hop data is CofCA's human-verified rewriting. The reference figures are GPT-4 at about 70% exact match factual versus about 53% on 2-hop counterfactual, and 36.3% correct reasoning chains (CofCA). We cannot reproduce its annotation budget. Instead, we compare against it in two ways:
- run the same reader models on CofCA and on our items matched by hop count, and check that the factual-minus-counterfactual gap is at least as large on our items;
- have annotators audit a random sample of our accepted items under CofCA's validity criteria.
Beating CofCA means an audited item-error rate within a few points of human-verified data, at an order of magnitude more items and at a cost per item recorded and compared.
For the verification step itself, the strongest competitor is a single-pass LLM judge. It reads the original paragraph, the intended edits and the rewrite, and declares the rewrite valid. Each verifier gets the same rewritten pool and the same model-call budget. We found no reported result on Lacuna for an LLM judge validating counterfactual passage rewrites. The closest reported evidence that round-trip failure tracks real meaning change is the autoformalization result: rules failing equivalence showed 1.4x to 2.5x more entailment-measured semantic drift, and scoped repair raised equivalence from 44.7% to 85.3% (Faithful Autoformalization). The round trip beats the judge if, on a human-labeled pool of rewrites, it achieves higher precision on accepted items at equal or higher acceptance yield.
Two ablations isolate the key ingredients. The first replaces proposition-set matching with whole-paragraph entailment between the intended and rewritten text, which tests whether atomicity matters. The second uses the same model family for the rewriter and the re-extractor, which measures canceling errors directly.
Finally, we evaluate three retrieval systems as subjects rather than as competitors: IRCoT, Chain-of-Retrieval Augmented Generation, and an agent with READ-GATE evidence-inspection invariants from Before Reasoning Can Fail. The question is whether their reported gains survive once memorized facts stop helping.
Why not just…?
Why not just swap entity names with fictional ones by string replacement? This is the real threat to the proposal. Find-and-replace is cheap and automatically consistent, and fictional names might remove most memory benefit. Two things count against it, though neither is settled.
- Renaming leaves relations intact. A model can still exploit the memorized fact that "the director of Inception was born in London" through the surviving title, date or role descriptions, so contamination leaks through the unchanged context.
- Renaming cannot toggle individual hops while keeping the chain coherent.
We include renaming as an explicit baseline arm. If its factual-minus-counterfactual gap and its per-hop override estimates match ours, the proposition-level machinery is not needed, and we will report that.
Why not just have an LLM rewrite passages and an LLM judge approve them? This is the single-pass judge baseline above. Our expectation, which the human-labeled pool will test, is that a holistic judge misses partial leakage, where the old fact survives next to the new one. Proposition-set matching tests for that case explicitly. The file gives no reported number either way. INVITE found that BLEU, ROUGE and even BERTScore failed to track truthfulness under distorted premises, which suggests holistic similarity signals are weak for exactly this kind of check.
Why not just generate synthetic graphs as DyVal does, or use only post-cutoff documents? DyVal-style graphs guarantee freshness but drop the retrieval over natural text that multi-hop systems are built for. Its striking result, models collapsing to near zero on generated tasks, concerns formal reasoning rather than evidence use. Post-cutoff news avoids memorized facts at first, but contamination returns as the documents age. It also cannot isolate individual hops, because the factual counterpart needed for a per-hop contrast does not exist. Both remain useful as complementary sanity checks, not substitutes.
Why not just ask reasoning models to think harder? The evidence points the other way. Before Reasoning Can Fail found that larger internal thinking budgets increased cases where agents answered without reading evidence. That is precisely the behavior $M_k$ measures, so we test thinking budget as an experimental factor rather than treat it as a fix.
Decisive Checks
The project succeeds if three things hold.
- Item validity. A blind human audit of a few hundred accepted items finds an error rate close to what CofCA's human-verified process would accept, and clearly lower than single-pass judging and same-family round trips.
- Contamination effect. The factual-minus-counterfactual accuracy gap on MuSiQue-derived items reproduces CofCA's direction and rough magnitude.
- Per-hop signal. $M_k$ varies meaningfully across hop positions and models, and correlates with closed-book knowledge of the specific hop proposition. That correlation is what shows the per-hop signal measures contamination rather than noise.
A further, more interesting outcome would concern the retrieval systems' advertised gains. If the gains of IRCoT, CoRAG or READ-GATE shrink on counterfactual items, part of their advertised improvement came from memory. If the gains hold, those methods are vindicated on contamination-resistant data. Either result is publishable, and the regenerable pipeline remains useful regardless.
References
- CofCA: A STEP-WISE Counterfactual Multi-hop QA benchmark
- BRIEF: Bridging Retrieval and Inference for Multi-hop Reasoning via Compression
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Faithful Autoformalization via Roundtrip Verification and Repair
- ♫ MuSiQue: Multihop Questions via Single-hop Question Composition
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
- Chain-of-Retrieval Augmented Generation
- Before Reasoning Can Fail: Pre-Evidence Procedural Failures in Agentic RAG
- INVITE: a Testbed of Automatically Generated Invalid Questions to Evaluate Large Language Models for Hallucinations