Question-answering systems that search in several steps, where the answer to one sub-question tells you what to search for next, have a characteristic way of failing. If one intermediate search comes back empty or irrelevant, the model does not stop. It fills in the missing fact from memory or guesswork and carries on confidently to a wrong final answer. Frontier models handle this badly. On multi-hop queries where only part of the needed evidence exists, GPT-5 gives an acceptable response only 28% of the time (CRUMQs). Models tuned for step-by-step reasoning abstain 24% worse than ordinary instruction-tuned models (AbstentionBench). The natural fix is to train abstention with reinforcement learning, meaning reward-driven fine-tuning on the model's own attempts. This works in single-turn math, where a tiered reward lifted a missing-premise behavior score from 8.66 to 51.73 (Ask, Condition or Abstain).
This note argues that whether such training learns the right thing depends on what the model can already tell apart when training starts. The idea is borrowed from learning with noisy labels. There, a short "warm-up" training phase has to produce features that separate clean examples from corrupted ones, or every later filtering step feeds on its own mistakes (DivideMix, PLS). The analogue in multi-step retrieval is this: before reinforcement learning begins, the model must already produce a per-step signal that separates "this intermediate fact came from the retrieved text" from "this fact was made up." Without that signal, the model can still learn to abstain, but it will abstain based on how the question is worded rather than on which step broke. The proposal makes this signal measurable, tests whether it predicts the outcome of reinforcement learning, and uses it to decide when warm-up is finished.
Core Claim
For each step $t$ of a multi-step retrieval chain, let $a_t$ be the intermediate answer the model writes, $q_t$ the sub-question it searched for, and $D_t$ the passages retrieved. Define the step's support score as
$$s_t = \log p_\theta(a_t \mid q_t, D_t) - \log p_\theta(a_t \mid q_t).$$
This is how much the retrieved passages raise the model's own probability of its intermediate answer, where $p_\theta$ is the model's probability under parameters $\theta$. A step grounded in the passages should score clearly positive. A step filled in from memory should score near zero. This plays the role of the "pseudo-loss" in PLS, which compared a model's guessed label with its prediction on a clean input to spot unreliable corrections.
We fit a two-component Gaussian mixture to the scores over a batch of chains, following the step DivideMix applies to per-sample losses. We then summarize how cleanly the two components separate, using the distance between their means scaled by their spread. Call this summary $S$. It needs no labels.
The claim has three parts:
- $S$, measured at the moment reinforcement learning begins, predicts how well the trained model points to the correct broken step and whether it behaves consistently on minimally different answerable/unanswerable pairs.
- It predicts these better than it predicts plain abstention rate.
- A warm-up stopped when $S$ crosses a threshold matches or beats a fixed-length warm-up at lower cost.
The claim is false if reinforcement learning started from a model with low $S$ reaches the same step localization and pair consistency as one started from a high-$S$ model.
How This Differs From What Exists
Three strands of prior work shape this proposal, each in a different role.
Evidence of the gap. Two benchmarks establish that the problem exists but offer no training method. CRUMQs builds realistic multi-hop queries that are partly unanswerable and finds that partial evidence is worse than none. MuSiQue composes multi-hop questions from single-hop ones so that no step can be skipped. Its MuSiQue-Full split pairs each answerable question with an unanswerable twin whose supporting context has been removed or modified. These twins are our main tool against shortcuts: a model that abstains based on question wording cannot tell the twins apart. Yang et al. show that such wording shortcuts are common in abstention for text-to-SQL.
The method we reuse and extend. Ask, Condition or Abstain supplies the tiered reward. Its authors restrict it to single-turn problems and report that plain "I don't know" is learned quickly while subtler behaviors appear slowly. That fast learning of plain abstention is exactly why we expect shortcut abstention in the multi-step setting. Warm Up Before You Train supplies the warm-up recipe. It distills long reasoning traces on Knights & Knaves logic puzzles (puzzles about truth-tellers and liars) before reward training, but it uses only solvable puzzles and a fixed schedule, and it never checks what the warm-up actually taught.
What we transfer, and what limits us. The noisy-label work (DivideMix, PLS, MOIT) supplies both the warm-up-dependency hypothesis and the mixture-model tool. It was developed only for image classification. Its known weakness is also ours: MOIT shows that at high noise the model memorizes corrupted examples during warm-up itself. We therefore expect that too much warm-up on fabricated chains could erase the separation we want, which is why stopping adaptively matters.
Research Program
Policy and retrieval loop. The policy is Qwen3-8B, the model used in Ask, Condition or Abstain. It runs an IRCoT-style loop (IRCoT) over the MuSiQue paragraph pools. At each step the model writes one reasoning sentence containing an intermediate answer. That sentence becomes the next search query, and retrieved paragraphs accumulate in the context. At every step the model may continue, answer, or abstain. When it abstains, it must name the step it believes is unsupported.
Warm-up arms. All arms are compared at matched token budgets:
- no warm-up;
- solvable Knights & Knaves traces, replicating Warm Up Before You Train;
- broken-chain Knights & Knaves traces, where one needed statement is deleted;
- in-domain supervised traces on MuSiQue-Full training data.
For the broken-chain arm, traces come from QwQ-32B, the teacher in the original warm-up paper. Because AbstentionBench suggests such a teacher will often invent the missing statement, we keep only traces that correctly identify the deleted premise. We know which premise that is, since we built the puzzle.
Reinforcement learning stage. This stage uses a multi-turn adaptation of the tiered reward:
- a correct answer on an answerable question earns full reward;
- abstaining on an unanswerable question and naming the right broken step earns nearly full reward;
- abstaining without the right step earns partial reward;
- abstaining on an answerable question earns a small penalty;
- any answer to an unanswerable question earns the largest penalty.
Following the finding in Ask, Condition or Abstain that such a mixture preserved standard reasoning performance, 30% of training questions are unanswerable.
Measurement. We log $S$ throughout warm-up on a held-out batch. Where gold labels exist, we also log the area under the ROC curve of the mixture posterior against true step support. The adaptive arm stops warm-up once $S$ plateaus above a threshold chosen on the synthetic puzzles alone. It never sees MuSiQue during this choice.
Scale of the sweep. The correlation study needs many reinforcement learning runs. It therefore runs on Qwen3-1.7B: four warm-up arms, three warm-up lengths, and three seeds. Qwen3-8B is used for the headline comparison.
Baselines
Naive reference. Qwen3-8B in the IRCoT loop with a prompt saying to abstain and name the broken step when evidence runs out. This shows what instruction alone achieves. AbstentionBench and CRUMQs, whose best reported figure is GPT-5 at 28% acceptable responses, suggest it will be weak on partial-context breaks.
Strongest existing method. The tiered reward of Ask, Condition or Abstain, applied directly in the multi-turn loop with no warm-up. In its original single-turn setting it raised Qwen3-8B's missing-premise behavior score from 8.66 under vanilla PPO to 51.73. It is the closest competitor because it already turns abstention into a trainable behavior; our claim only adds a condition on the starting point. For fairness, it gets the same data, the same reward, the same reinforcement learning steps, and the warm-up arms' token budget added as extra reinforcement learning steps, so total compute is equal. We beat it if we improve twin-pair consistency and broken-step localization on MuSiQue-Full by more than the seed-to-seed spread, without lower answerable F1. For multi-step abstention on MuSiQue-Full or CRUMQs we found no reported result on Lacuna for this method or for IRCoT. Both are therefore run by us. The IRCoT paper reports gains of up to 15 F1 on answerable data only.
Training-free gate. The same support score with a mixture model fitted at inference time, used as a gate on the untrained model. It abstains whenever a step's posterior probability of being fabricated exceeds one half. This isolates whether the signal alone, without training, is enough.
Verifier. An entailment-model check in the style of Can NLI Models Verify QA Systems' Predictions?, applied to each step's sentence and passages. That paper reports rejecting 78.5% of unanswerable SQuAD 2.0 questions zero-shot, though on single sentences.
Key ablation. Fixed-length versus adaptively stopped warm-up at matched data, plus solvable versus broken-chain puzzles. This isolates whether the benefit comes from separability or from warm-up in general.
Why not just…?
Why not just prompt the model to say "I don't know"? Existing evidence says prompting fails specifically where we care. CRUMQs reports 28% acceptable responses for GPT-5 on partial-context multi-hop queries, and AbstentionBench shows reasoning models abstaining 24% worse. We still run prompting as the naive reference, broken down by which step fails.
Why not just check the final answer with a verifier? A check after the fact cannot identify which step broke once later searches have been steered by a fabricated fact. The entailment approach in Can NLI Models Verify QA Systems' Predictions? also explicitly works only when the evidence is a single sentence. We test this directly with the per-step verifier baseline, measuring both localization and false rejections of valid multi-sentence steps.
Why not just run the tiered reward without any warm-up? This is the real threat. Plain abstention is learned fast (Ask, Condition or Abstain), so reinforcement learning alone may do fine. The compute-matched no-warm-up arm settles it. If it matches on twin consistency and localization, our dependency claim is refuted and we report that. The correlation sweep tells us whether any gap tracks $S$ or merely tracks extra training.
Why not just fine-tune on MuSiQue-Full unanswerable examples instead of synthetic puzzles? In-domain supervised warm-up may well produce higher $S$ than puzzles, and it is included as an arm. The claim concerns separability, not puzzles. If in-domain data wins, the conclusion becomes "warm up on in-domain broken chains until $S$ plateaus." The puzzle arm then matters only as the cheap option for domains that lack labeled unanswerable data.
Decisive Checks
Primary evaluation. The main test is MuSiQue-Full. We measure:
- twin-pair consistency: answering the answerable twin and abstaining on its unanswerable twin;
- localization accuracy of the named broken step;
- abstention accuracy broken down by the position of the break, since breaks at the last step give the most partial context;
- answer F1 on MuSiQue-Ans, to confirm answerable performance is kept.
Transfer. We report acceptable-response ratio on CRUMQs and abstention recall on AbstentionBench using its LLM judge. That judge agrees with humans 88% of the time. Answer F1 on HotpotQA and 2WikiMultihopQA checks that multi-hop answering does not degrade.
The dependency test. Across the 36 small-model runs, we compute the rank correlation between $S$ at the start of reinforcement learning and final twin consistency, and compare it with the correlation between $S$ and raw abstention rate. The hypothesis predicts a strong first correlation and a weaker second one. We also check for the high-noise collapse seen in MOIT: if $S$ falls again during long broken-chain warm-up, fixed-length schedules should underperform the adaptive stop. If neither pattern appears, warm-up does not govern what abstention training learns in this setting, and the simpler direct reinforcement learning approach should be preferred.
References
- Evaluating Retrieval-Augmented Generation Systems on Unanswerable, Uncheatable, Realistic, Multi-hop Queries
- AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions
- Ask, Condition or Abstain: Reinforcement Learning for Missing-Premise Reasoning
- DivideMix: Learning with Noisy Labels as Semi-supervised Learning
- Is your noise correction noisy? PLS: Robustness to label noise with two stage detection
- ♫ MuSiQue: Multihop Questions via Single-hop Question Composition
- Towards Unbiased Evaluation of Detecting Unanswerable Questions in EHRSQL
- Warm Up Before You Train: Unlocking General Reasoning in Resource-Constrained Settings
- Multi-Objective Interpolation Training for Robustness To Label Noise
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
- Can NLI Models Verify QA Systems' Predictions?