Dropping Memorized Steps: Counterfactual Loss Truncation for Multi-Hop Reasoning Traces

Language models are increasingly taught multi-hop reasoning (answering questions whose evidence is spread across several documents) by fine-tuning on reasoning traces generated by a stronger model or a search procedure. These traces are kept if they reach the right final answer. A trace can reach the right answer while one of its steps states a fact the provided documents never supported, because the fact was recalled from pre-training memory. Standard fine-tuning then teaches the student to recall rather than read. The consequence shows up on counterfactual benchmarks, where the documents contradict what the model memorized. There, GPT-4's exact match falls from about 70% to about 53% on 2-hop questions, and only 36.3% of its reasoning chains are fully correct (CofCA).

Summarization researchers already faced a version of this problem. Their training references contain facts absent from the source, and they found a fix: skip the training tokens the model finds hardest to predict, a technique called loss truncation (Improved Natural Language Generation via Loss Truncation). Our central observation is that this fix points the wrong way for reasoning traces. A step copied from memory is not hard to predict. It is easy, often the easiest step in the trace. We therefore propose to truncate steps by a different signal: how much the model's confidence in a step drops when the supporting document is edited to say something else. A step grounded in the document should become improbable once the document contradicts it. A memorized step stays probable no matter what the document says. Removing low-sensitivity steps from the training loss should produce students that read their context, and a counterfactual test set should show it.

Core Claim

Let $s_k$ be the $k$-th reasoning step of a training trace and $C$ the provided documents. Let $C'_k$ be a copy of $C$ in which the entity that step $k$ asserts has been replaced, inside its supporting paragraph, with a type-matched alternative. We define the step's grounding score as

$$ g_k = \log p_\theta(s_k \mid C, s_{<k}) - \log p_\theta(s_k \mid C'_k, s_{<k}), $$

where $p_\theta$ is the student being trained. A large $g_k$ means the step's probability depends on the evidence. A $g_k$ near zero means the model would say the same thing whatever the document said. During fine-tuning we keep a running estimate of the $c$-quantile of $g_k$ across recent steps, in the style of loss truncation, and we mask the loss on steps that fall below it.

We add one safeguard. Steps whose claimed entity cannot be found in any provided paragraph cannot be edited, so they are masked outright as unsupported. This case is the classic high-loss hallucination that ordinary truncation targets.

The claim is that masking by $g_k$ reduces the gap between factual and counterfactual accuracy more than three alternatives: masking by raw loss, filtering by final-answer correctness, or filtering by an entailment classifier. It should do so without discarding whole traces and without lowering accuracy on factual benchmarks beyond a small tolerance.

Where This Sits

Three lines of prior work each supply part of the picture.

The summarization work supplies the method and its hidden assumption. Improved Natural Language Generation via Loss Truncation drops high-loss examples on Gigaword and E2E and raises human-rated factuality to 3.79, above human references at 3.63. Training Dynamics for Text Summarization Models shows that unsupported tokens are learned late and with high loss, and that dropping them cuts XSum sentence error rate from 0.67 to 0.59. Both papers assume that noise is hard to predict. That holds when the noise is foreign to the model. It should fail when the noise is the model's own prior knowledge, which is the regime that contamination creates. We keep their adaptive-quantile machinery and change the quantity it thresholds.

The multi-hop training work supplies the setting and the noisy traces. GRAT collects 11,000 tree-search traces that end in correct answers and fine-tunes Llama3-8B on them with plain cross-entropy, raising 2WikiMultiHopQA accuracy from 60.6% to 66.8%. CARE teaches models to quote evidence from context with special tokens and reinforcement learning, and reports a +15.29% average F1 gain over LLaMA-3.1 8B, including on CofCA. Neither method asks whether an individual step in a correct trace was read or recalled. We change only the training loss, so our method can sit on top of either.

The shortcut and contamination work motivates the proposal and supplies the evaluation. MuSiQue shows that correct answers often come from disconnected reasoning, meaning shortcuts that skip intermediate hops, and measures this with its DiRe score. Robustifying Multi-hop QA through Pseudo-Evidentiality Training is the closest methodological relative. It removes sentences, watches how confidence changes, and uses the result to split examples into evidential and non-evidential sets for an extractive RoBERTa model. Our method differs in three ways. It works on individual generated steps rather than whole examples. It edits the evidence rather than deleting it. It applies to autoregressive reasoning traces. CofCA is purely an evaluation resource. We use its idea, counterfactual edits, as a training-time signal for the first time.

One piece of evidence cuts against us. CofCA reports that open models suffer knowledge conflicts, where memory overrides context. If memory dominates strongly enough, the student's $g_k$ may be near zero even for grounded steps, and the score would carry no signal. We test this directly in the first experiment.

Research Program

We build training traces from the MuSiQue-Ans and 2WikiMultihopQA training splits. Both provide gold decompositions with intermediate answers and supporting paragraphs. A Llama3-70B teacher generates interleaved retrieve-and-reason traces in the style of IRCoT over the gold paragraphs plus distractors. We keep traces whose final answer is correct, as GRAT does.

Each step is aligned to a paragraph by string-matching its asserted entity. We then construct $C'_k$ by swapping that entity for another entity of the same type drawn from the training corpus. Scoring costs one extra forward pass per step, which is cheap next to generating the traces.

The student is Llama-3.1 8B, which matches the backbone CARE reports on. Before any evaluation we remove from training every passage that CofCA's factual control set draws on.

Three analyses come first, because they decide whether the method is needed at all. We hand-annotate 500 steps as grounded, recalled-but-true, or unsupported, and plot per-step loss and $g_k$ for each class. If recalled steps turn out to be high-loss, ordinary step-level loss truncation already solves the problem. In that case we would report this and stop.

We also track how $g_k$ evolves over training. The goal is to see whether memorized steps separate from grounded ones early, mirroring the copy-first dynamics described in Training Dynamics for Text Summarization Models. If they do, that tells us when to switch truncation on.

Finally, we sweep the quantile $c$, since sensitivity to this hyperparameter is a known weakness of loss truncation.

Baselines

The naive reference point is the untuned Llama-3.1 8B prompted with chain-of-thought over the same paragraphs. It tells us how much any fine-tuning helps.

The main competitor is outcome-filtered fine-tuning: plain cross-entropy on every correct-answer trace, the recipe of GRAT. GRAT reports 60.6% to 66.8% accuracy on 2WikiMultiHopQA for Llama3-8B. Our traces come from a different generator, so we reproduce this baseline on our own trace pool with the same data, steps, and learning-rate search rather than comparing against GRAT's published number.

We also compare against CARE, the strongest context-fidelity method in the file, with its reported +15.29% average F1 over base LLaMA-3.1 8B. We run it from its released recipe where possible. We also apply our truncation to its fine-tuning phase, to test whether the two methods add up.

Step-level loss truncation masks the highest-loss steps at the same drop rate as our method. It is the direct transfer of Improved Natural Language Generation via Loss Truncation, and it is the ablation that isolates our key ingredient: which signal decides what gets dropped. A random-step mask at the same drop rate controls for the mere effect of seeing fewer tokens.

We count our method as having won if two conditions hold. On CofCA, the gap between factual and counterfactual accuracy, together with joint step-and-answer correctness, must improve over outcome-filtered fine-tuning and over loss truncation by more than the variation across three seeds. Factual MuSiQue and 2Wiki F1 must fall by no more than one point.

Why not just…?

Why not just scale up the teacher or the student? Scale does not remove the memory-over-context failure. GPT-4 is far larger than our student and still manages only 36.3% fully correct chains on CofCA (CofCA). IRCoT shows that a 3B model with a better reasoning procedure beats a 175B model without it. A larger teacher also recalls more facts, so it may produce more memorized-but-correct steps rather than fewer. Our annotation study counts such steps for the 8B and 70B teachers directly.

Why not just filter each step with an entailment classifier before training? This is the step-level version of the SummFC ensemble in Questioning the Validity of Summarization Datasets and Improving Their Factual Consistency. It is a real threat, and we include it as a baseline. It has a structural blind spot. When a recalled fact happens to be true and appears in the paragraph, an entailment model will pass it, even though the student produced it without reading. Such steps are harmless on factual benchmarks and harmful on counterfactual ones. The decisive comparison is therefore the counterfactual gap, not factual F1. If the entailment filter matches us on CofCA, the cheaper filter wins and we will say so.

Why not just train on counterfactually edited contexts directly? Training on edited contexts removes the possibility of recall altogether, and for 2Wiki's rule-based questions edited answers can be propagated automatically. This is the strongest alternative. It requires every downstream step and the final answer to be relabeled consistently. That is easy for structured 2Wiki questions and error-prone for free-form MuSiQue traces, which is why CofCA used human annotators. Our method needs the edited context only to score a step, not to supply a new label. We run the augmentation baseline on 2Wiki, where it is feasible, and we also run it combined with our method. If augmentation alone matches us there, our contribution shrinks to the free-form case, and we will report the result that way.

Proposed Experiments

We evaluate on MuSiQue-Ans, 2WikiMultihopQA, and HotpotQA using exact match and F1, and on MuSiQue with the DiRe score to measure disconnected reasoning. CofCA is the primary test. We report exact match on its factual and counterfactual halves, the gap between them, and joint performance at 2, 3, and 4 hops.

We expect gains to grow with hop count, because every memorized step in a longer chain is another chance for recall to override context. Because reference-based F1 can fall when a model stops reproducing what the references contain, a trade-off already documented for summarization, we also report human-judged step grounding on 300 sampled test traces.

The decisive checks are three. First, the annotation study must show that recalled steps are low-loss but have low $g_k$. Second, the ablation that matches drop rates must show that $g_k$ beats loss and random masking on the CofCA counterfactual gap. Third, the method must beat both the entailment filter and counterfactual augmentation on the free-form MuSiQue-derived part of CofCA. If the first check fails, the premise is wrong. If the third fails, the method is redundant with simpler tools.