Counterfactual Targets for Multi-Hop Data Mixture Search

Fast data-mixing methods decide how much web text, code, math, Wikipedia, and so on goes into a language model's training data. They work by training a small stand-in model, called a proxy, and adjusting the proportions so that training on each source moves the proxy toward good performance on a target validation set. These methods are now cheap enough to run in about a GPU-hour. That makes the choice of target the decision that matters most, and it is rarely examined.

For multi-hop reasoning (answering a question by chaining two or more facts, such as "the birthplace of the director of film X"), the standard validation sets come from Wikipedia questions. Large models can often answer these from memorized facts without composing anything. A mixture optimizer aimed at such a target will rationally upweight data that improves recall of Wikipedia facts, and any source that happens to contain leaked benchmark questions. It will not upweight data that teaches composition.

We propose to replace the target. Instead of benchmark questions, the optimizer would aim at procedurally generated counterfactual multi-hop items: every bridge fact is supplied in the context, and the facts contradict or are absent from world knowledge, so memory cannot help and only chaining can. The claim to test is that this one substitution changes which mixture a cheap optimizer picks. The resulting mixture should transfer better to memorization-resistant multi-hop benchmarks at larger scale, and it should stop rewarding deliberately planted contamination. All of this should come at essentially no added search cost.

Core Claim

The intersection is covered piecemeal but not in this combination. FastMix turns mixture search into gradient descent on per-source loss weights using one proxy run. DOGE and Dynamic Gradient Alignment upweight sources whose training gradients point in the same direction as the gradient of a target loss. None of the three asks whether the target can be satisfied by memorization.

On the evaluation side, the picture is consistent:

  • CofCA rewrites Wikipedia entities into fictional ones. It reports that GPT-4's exact match on 2-hop questions falls from about 70% on factual data to about 53% on counterfactual data, with only 36.3% of reasoning chains fully correct.
  • MuSiQue filters out questions that can be answered by skipping a hop.
  • DyVal generates reasoning problems from graphs and finds that depth (more sequential steps) hurts far more than width (more parallel premises).

All three are evaluation instruments only. None is used to choose training data.

Front-Loading Reasoning shows why the pre-training mixture is the right place to act. Reasoning absent from pre-training is not recovered by doubling the fine-tuning data. That study, however, used a fixed 20% reasoning share and did not search over mixtures.

Our contribution is narrow and testable: a cheap counterfactual multi-hop target for existing optimizers, plus the experiments that show whether the target, rather than the optimizer, decides what gets learned.

We define the counterfactual target loss as

$$L_{cf}(\theta) = -\frac{1}{N}\sum_{i=1}^{N} w_{h_i}\,\log p_\theta\big(a_i \mid c_i, q_i\big),$$

where the symbols are:

  • $\theta$ is the proxy's parameters.
  • $q_i$ is a multi-hop question.
  • $c_i$ is a context containing the counterfactual bridge facts plus distractors.
  • $a_i$ is the answer, which is implied only by chaining the context facts.
  • $h_i$ is the number of hops in item $i$.
  • $w_{h}$ is a per-depth weight that we set to keep deeper items from being drowned out, motivated by DyVal's depth finding.

Because the loss is a likelihood rather than an accuracy, it gives a usable signal even for proxies too small to answer correctly. This is the same reason RBRIDGE moves to likelihoods. Unlike RBRIDGE, our signal does not depend on a teacher model's possibly contaminated reasoning traces.

Research Program

Target generator. We first build a generator in the spirit of DyVal, but for text. It samples small knowledge graphs of fictional or attribute-swapped entities, writes each edge as a one-sentence fact, and composes 2-, 3- and 4-hop questions by walking paths.

We apply MuSiQue-style disconnection checks: an item is discarded if a model given only a subset of the facts can still answer it. We also include "conflict" items, where the context states a fact that contradicts a well-known real fact. On these items memory actively hurts.

The generator is cheap and unlimited. This removes CofCA's limit of 1,800 human-built samples and lets us redraw fresh validation items at every search step.

Optimizers. We then run two existing optimizers unchanged except for the target:

  • FastMix's gradient-based search over static per-source weights.
  • DOGE's gradient-alignment reweighting.

Both run on fixed source partitions: the seven SlimPajama domains used by DOGE, and The Pile's 22 domains used by FastMix and DoReMi.

Proxies are 60M to 124M parameters, the range at which DOGE found stable weights. Following Can Small Training Runs Reliably Guide Data Curation?, proxies are trained at very small learning rates. That paper reports this raises cross-scale rank correlation from below 0.75 to above 0.92.

Analysis. The analytic part asks why the mixtures differ. For each source, we measure how much training on it alone moves two quantities:

  • the factual-target loss, which uses real Wikipedia multi-hop items, and
  • the counterfactual-target loss.

Call the difference a source's "recall share": the part of its apparent multi-hop value that comes from memorized facts. We expect encyclopedic sources to have a high recall share. We expect sources dense in explicit procedural or relational text, such as code, math, and argumentative prose, to have a low one.

Baselines

Uniform mixture. The naive reference point is uniform sampling over sources, trained with identical tokens and architecture. The literature consistently finds it beatable: DGA reports 28.0% versus 26.1% for uniform on MMLU at 750M parameters, and MergeMix reports 58.4% versus 54.2% for uniform at 8B. Beating uniform is necessary but says little.

Fixed reasoning share. The heuristic reference is a fixed reasoning-heavy upweighting in the style of Front-Loading Reasoning. We assign a 20% share to the sources a human would call reasoning-rich, to check whether search adds anything over that recipe.

Same optimizer, standard target (strongest competitor). The strongest competitor is FastMix with the same proxy, the same source partition and the same search compute, but with a conventional multi-hop target: HotpotQA development items in the distractor setting. This is the most efficient published optimizer, and it would be the obvious way a practitioner would "optimize for multi-hop."

FastMix reports a 48.2 average over 14 benchmarks at 1B parameters in 1.3 GPU-hours, against 47.2 for RegMix and 47.5 for CLIMB. I found no reported result on Lacuna for FastMix, or any mixture optimizer, aimed at a multi-hop target. We will therefore run this baseline ourselves at matched budget. "Matched budget" means the same number of proxy steps, the same learning-rate sweep, and the same number of validation items per step.

Same optimizer, MuSiQue target. This variant aims at a shortcut-filtered real-entity target. It is the closest thing to our idea that needs no new generator.

RBRIDGE-style target. This variant uses teacher-written reasoning traces for HotpotQA as the target likelihood. It represents the current best cheap reasoning proxy.

DoReMi. DoReMi is the task-agnostic option. It uses no target and minimizes worst-case excess loss across sources.

Ablations of the key ingredient. The key ingredient is that the target cannot be satisfied by memory. We isolate it in two ways:

  • Paired targets: the same generated graphs rendered once with real entities and true facts, and once counterfactually. The structure is identical and only the possibility of using memory changes.
  • Depth weighting: the full target with $w_h$ uniform versus depth-reweighted.

What counts as beating the strongest baseline. The counterfactual-target mixture must beat the HotpotQA-target and MuSiQue-target mixtures on CofCA and MuSiQue at the 1B target scale, by more than the seed-to-seed spread across three training seeds. It must also stay within one point of their average on FastMix's general 14-benchmark suite.

Why not just…?

Why not just aim the optimizer at MuSiQue, which already filters shortcuts? This is the most serious threat. MuSiQue's disconnection filtering removes questions that can be answered by skipping a hop. It does not stop a model from answering each hop from memorized Wikipedia facts. CofCA's motivation is precisely that multi-hop benchmarks, MuSiQue included, are heavily represented in pre-training data and invite memory shortcuts. Still, MuSiQue may turn out to be good enough. The MuSiQue-target arm is in the main comparison for that reason. If its mixture matches ours on CofCA and on held-out generated items, we will report that the generator is unnecessary and that the finding is "use shortcut-filtered targets."

Why not just decontaminate the target with n-gram or loss-based filters? These filters address the wrong problem. Verbatim filtering removes copied questions, but CofCA's results show that models bypass hops through factual associations even without verbatim leakage. Loss-based detectors are also unreliable on reasoning-trained models: On The Fragility of Benchmark Contamination Detection in Reasoning Models reports that ten detectors fall to 50–58% AUROC after training on chain-of-thought data. Our planted-contamination experiment below tests directly whether a filtered factual target still rewards the leaking source.

Why not just use a fixed reasoning-heavy heuristic, as in Front-Loading Reasoning? That study shows that more reasoning data in pre-training helps. It does not show which sources supply composition rather than recall, and its 20% share was never optimized. The heuristic arm measures whether search adds value. If it ties our mixture, the honest conclusion is that the target matters less than the coarse reasoning share. We would report that.

Why not just fix multi-hop at inference time with interleaved retrieval such as IRCoT? IRCoT alternates writing reasoning steps with retrieving documents. This is complementary to our proposal, not a substitute: it depends on the model writing correct intermediate steps. MuSiQue reports roughly a 30-point F1 drop for single-hop models once shortcuts are removed. We will evaluate all target models both with and without an IRCoT-style loop, to check that mixture gains persist under retrieval scaffolding.

Proposed Experiments and Decisive Checks

Main study. Each search arm produces a mixture. We train each mixture into 1B-parameter models on The Pile sources for a fixed token budget, with three seeds.

Every model then receives the same light, standardized multi-hop fine-tuning on a fixed generated training split with entities disjoint from all evaluation sets. We do this because Front-Loading Reasoning argues that pre-training sets the ceiling. Fine-tuning is used only to read that ceiling out consistently across mixtures.

We evaluate on four groups:

  • Memorization-resistant multi-hop: CofCA by hop count, measured with exact match and step-wise chain correctness.
  • Real-entity multi-hop: MuSiQue-Ans answer and support F1 (human performance is reported at 78.0 answer F1), HotpotQA distractor-setting F1, and 2WikiMultihopQA.
  • Held-out generated items: graph families and depths not used during search, including 5-hop items.
  • General capability: FastMix's general benchmark suite, to catch regressions.

Decisive check 1: planted contamination. We inject HotpotQA and 2WikiMultihopQA development questions and answers into one candidate source at low rate, and leave everything else unchanged. Under the reasoning of the Gap analysis, gradient alignment with a HotpotQA target should push weight toward the poisoned source.

The counterfactual target should be indifferent to it, because leaked real facts cannot lower loss on fictional chains. If the counterfactual-target optimizer also upweights the poisoned source, the core mechanism is wrong.

Decisive check 2: recall share. The recall-share analysis must predict the direction of the mixture shift. Sources with a high recall share should lose weight under the counterfactual target relative to the factual target.

The paired-target ablation must show the same shift when only the possibility of using memory changes. If the shift appears with counterfactual items but not with paired structure-matched factual items, then the effect comes from memory and not from the generator's question style.

Decisive check 3: transfer. The gain must transfer across scale and across benchmarks the search never saw. Concretely, the CofCA and MuSiQue advantage of our mixture over the HotpotQA-target mixture must hold at both 1B and a smaller 400M target. The total search cost, including item generation, must stay within a small multiple of FastMix's reported 1.3 GPU-hours.

Interpreting a null result. If the counterfactual target yields no gain, the paired results still say something useful. They show whether cheap mixture search cannot see composition at all at proxy scale, or whether the standard benchmark target was already harmless.