Systems that answer multi-document questions by alternating reasoning and retrieval can fail silently. A "hop" is one round of writing a reasoning step, searching a corpus with it, and reading what comes back. Sometimes a chain of hops reaches the right answer through a shortcut: it skips a hop, or retrieves a passage it never uses, while the final answer still looks correct. If such systems are trained only on final-answer correctness, the shortcut chains are rewarded as much as the sound ones. The problems then show up at depth, where shortcuts stop working and each bad hop feeds the next. The obvious fix is to label every intermediate hop, but step-level human labels are the expensive part.
We propose to learn a hop-level verifier, meaning a model that scores whether each retrieval-and-reasoning step is sound, using only final-answer labels. The final answer is treated as a coarse "bag" label over the hops of a chain, as in weakly supervised object detection. Structural constraints then stop credit from collapsing onto a single decisive hop. These constraints are adapted from the vision work that solved the same collapse for image regions. The verifier then guides search, backtracking, and training of the reasoning policy. The main test is on MuSiQue, whose gold decompositions and supporting paragraphs are held out and used only for evaluation. That lets us check directly whether answer-only training recovered correct hop-level judgments.
Core Claim
Training on final answers alone has the same failure as training an object detector on image-level labels. The detector learns to fire on the most discriminative part (a cat's face) and ignores the rest of the object. The reasoning model learns to credit the one hop that makes the answer guessable and ignores whether the chain is connected. Vision fixed this with two tools. WELDON (WELDON) pools both the strongest supporting and the strongest contradicting regions. Weakly Supervised Detection with Posterior Regularization adds prior constraints such as symmetry and mutual exclusion on the distribution over hidden regions, which raised detection from 22.7% to 26.4% mAP on Pascal VOC 2007 without box labels.
Our claim is that the analogous constraints on hops will do two things when training from answer labels only. First, they will produce a verifier whose per-hop judgments agree with gold supporting facts far better than an answer-only baseline. Second, a retrieval-reasoning system guided by that verifier will degrade more slowly as the number of required hops grows. MuSiQue's evidence that multi-hop models pervasively use "disconnected reasoning" shortcuts (MuSiQue) motivates the first claim. The depth cliffs reported by DyVal and PRONTOQA-OOD motivate the second.
Relation to Closest Work
Two papers each do half of this. AutoPRM (AutoPRM) builds step-level supervision without humans by decomposing questions into subquestions and checking sub-answers against the final answer. It works only closed-book, lists retrieval integration as future work, and itself warns that answer-only feedback can reward invalid chains. We supply the method it lacks for that case, and we add retrieval.
MoPo (MoPo) also uses posterior regularization in multi-hop retrieval. It distills a teacher retriever that sees the answer into a student retriever that sees only the query. It improves the retriever only, and it relies on LLM summaries written from gold passages. Our use of posterior regularization differs in kind. MoPo matches a student to a teacher. We impose structural constraints on a hidden credit assignment over hops, and we use no gold passages at training time.
IRCoT (IRCoT) supplies the inference loop we build on: interleaved reasoning steps, each used as a search query. It has no verifier and no backtracking, which is exactly the weakness the survey literature on cascading hallucination in retrieval-augmented generation highlights (Attribution Techniques for Mitigating Hallucinated Information in RAG Systems: A Survey). Adaptive-retrieval methods such as DRAGIN (Su et al. (2024)) decide when to retrieve from token uncertainty, and Auto-RAG (Yu et al. (2024)) fine-tunes planning. Neither learns to judge whether a completed hop was sound.
Research Program
Sampling chains. We start from an IRCoT-style policy, using the same base model family as AutoPRM (LLaMA-2-7B) for comparability. For each training question in MuSiQue, HotpotQA and 2WikiMultihopQA, we sample many hop chains. Each chain is labeled only by whether its final answer matches.
Latent credit and the objective. Each hop $h_t$ in a chain gets a latent soundness variable $z_t \in \{0,1\}$. The verifier $p_\theta(z_t \mid \text{question}, h_{1:t})$ is trained with a soft multiple-instance objective: a correct chain should have all its hops sound, while an incorrect chain has at least one unsound hop. Following Bilen et al., the posterior over $z$ is fit subject to constraints:
$$\min_{\theta,\, q}\; \mathrm{KL}\big(q(z) \,\|\, p_\theta(z \mid \text{chain}, y)\big) \quad \text{s.t.}\quad \mathbb{E}_{q}[\phi_k(z, \text{chain})] \le b_k \ \ \text{for each constraint } k,$$
Here $y$ is the final-answer correctness label, $q$ is the constrained posterior over hop soundness, $\phi_k$ is a constraint feature, and $b_k$ is its slack.
The three constraints. Each $\phi_k$ is computed automatically, without gold labels.
The first is an erasure-necessity constraint, the counterpart of WELDON's contradictory evidence. If deleting hop $t$'s retrieved passage and re-answering leaves the answer model's confidence unchanged, $q$ may not assign that hop positive credit. Hops that contribute nothing are not credited even in correct chains.
The second is a connectivity constraint that targets MuSiQue-style disconnected reasoning. Hop $t$'s search query must contain an entity or value that first appeared in the passages of hops $1..t-1$. A chain that retrieves the answer passage directly from the question, without passing through the bridge entity, cannot be fully credited on a question the decomposer judges to be multi-hop.
The third is a mutual-exclusion constraint, the counterpart of Bilen et al.'s $PR^{me}$. Two hops credited as sound should not retrieve the same passage or resolve the same entity. This blocks chains that pad themselves by repeating one discriminative hop.
Using the verifier. We use it in two ways. At inference it runs a small beam over hops: candidate hops scoring below a threshold are pruned and the policy backtracks to the previous hop. At training it serves as a per-hop reward for expert iteration of the policy, as in AutoPRM.
Controlled-depth evaluation suite. Public multi-hop benchmarks stop at four hops. We therefore generate a suite in the style of PRONTOQA-OOD's fictional ontologies. Each fact is rendered as a separate short document in a retrieval corpus. We plant GSM-DC-style off-path distractor documents (How Is LLM Reasoning Distracted by Irrelevant Context?), and required depth runs from 2 to 8. Fictional entities rule out the memorization that DyVal showed inflates static-benchmark scores. The suite also lets us test an open tension in the literature. Saparov et al. found simple, diverse demonstrations generalize better to deep proofs in few-shot prompting, while GSM-DC found training on hard distractors generalizes better in fine-tuning. We will compare verifier training on shallow versus deep chains to see which pattern holds here.
Baselines
Naive reference points. The weakest reference is one-step top-$k$ dense retrieval followed by answering. On full-wiki HotpotQA this style of system is known to be weak: the original baseline reported 17.73 joint F1 (HotpotQA). The second "do nothing" point is plain IRCoT with greedy decoding and no verifier, run with our base model, retriever and corpus. It isolates the value of adding any verifier at all.
Strongest external method. On HotpotQA this is the MoPo pipeline, which reports 64.3 F1 against 62.9 for BeamAggR and 56.2 for IRCoT (MoPo). MoPo improves the retriever and our method improves hop judgment, so the two are complementary. We will run our verifier-guided search both on top of a standard retriever and on top of a MoPo retriever. Beating MoPo means exceeding 64.3 F1 on HotpotQA with the MoPo retriever plus our verifier, at no more inference calls than the IRCoT pipeline MoPo compared against.
MuSiQue is our primary target because it resists shortcuts, and here the file gives no comparable baseline number. There is no reported result on Lacuna for IRCoT's or MoPo's MuSiQue answer F1; the only figure is human performance at 78.0 answer F1 and 93.9 support F1 (MuSiQue). We will therefore reproduce IRCoT on MuSiQue ourselves, with its own released settings, and report it alongside our system.
Answer-only verifier (key ablation). This is the strongest competitor on the actual question. It uses identical chains, identical base model, and identical training steps and search budget, but the verifier is trained from the final-answer label with no constraints. This is essentially expert iteration or rejection-sampling fine-tuning on correct chains. A second ablation drops each constraint in turn to show which one carries the effect.
AutoPRM-with-retrieval. Here we apply AutoPRM's decomposition-based step feedback to the retrieval loop, using the granularity setting ($\epsilon = 0.8$) it reported worked best. This tests whether sub-answer checking already suffices.
Fair comparison and what counts as winning. All trained variants get the same number of sampled training chains and hyperparameter trials. All inference variants are matched on total LLM calls. Our method wins if three things hold. Its hop-verifier agreement with MuSiQue gold supporting paragraphs (area under the ROC curve) clearly exceeds the answer-only ablation's. Its answer and support F1 on MuSiQue 3- and 4-hop questions improve at matched calls. And its accuracy falls with depth on the synthetic suite at a visibly smaller slope.
Why not just…?
Why not just keep the chains that got the right answer and fine-tune on them? This is the real threat. If shortcuts are rare in practice, answer filtering alone may produce a nearly identical verifier. MuSiQue's finding that single-hop models lose 30 F1 points once disconnection filters are applied suggests shortcuts are common on easier data. AutoPRM also flags reward tampering from answer-only feedback. Still, neither result shows that filtering training chains fails. The answer-only ablation settles it directly. We will measure the "disconnected-correct" rate, meaning chains with correct answers that miss a gold supporting paragraph, in both verifiers' top-ranked chains. If our constraints do not reduce that rate, the idea is wrong.
Why not just use self-consistency at test time? Sampling many chains and voting helps, for example a 17.9-point absolute gain on GSM8K with PaLM-540B (Self-Consistency). But it multiplies inference cost and relies on answers from a small closed set, whereas multi-hop answers are open-ended spans. More basically, voting cannot tell a sound chain from a shortcut chain that reaches the same answer. We compare at matched LLM calls and report both answer F1 and support F1. The support measure is where we expect voting to fall short.
Why not just have a strong LLM label each hop? This is a cheap substitute for human annotation and is plausible. Its limit is DyVal's report that GPT-4 still produced unsubstantiated responses in 33.8% of its failures on deep graphs, which suggests that judging deep chains is itself unreliable. We include a prompted-judge verifier, matched on cost, and score it against MuSiQue gold supports and on the synthetic suite, where gold is exact. If it matches our verifier, the practical conclusion is to use the judge, and we will say so.
Why not just use a bigger model with a long context? IRCoT showed a 3B model with interleaved retrieval beating 175B GPT-3 with one-step retrieval. PRONTOQA-OOD showed depth breakdowns from depth 2 to depth 5 across all tested scales. The synthetic suite, run with a larger model given the whole corpus in context, will show whether scale alone flattens the depth curve.
Decisive Checks
Three results would confirm the proposal. First, on MuSiQue dev, the constrained verifier should agree markedly better with gold supporting paragraphs than the answer-only verifier, even though neither saw any gold at training time. Second, at matched LLM calls, verifier-guided IRCoT should raise MuSiQue support F1 and answer F1, with the gap growing from 2-hop to 4-hop questions. Third, on the synthetic suite from depth 2 to 8 with 1 to 15 distractors, the accuracy-versus-depth curve should fall with a flatter slope than the IRCoT and answer-only curves.
If the first check fails, the constraints are not recovering hop soundness, and any downstream gain is incidental. If the first holds but the third does not, verification helps at the depths seen in training but does not stop decay at greater depth. That would still be a useful negative result about the limits of weak step supervision.
References
- WELDON: Weakly Supervised Learning of Deep Convolutional Neural Networks
- Weakly Supervised Detection with Posterior Regularization
- ♫ MuSiQue: Multihop Questions via Single-hop Question Composition
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examples
- AutoPRM: Automating Procedural Supervision for Multi-Step Reasoning via Controllable Question Decomposition
- Momentum Posterior Regularization for Multi-hop Dense Retrieval
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
- Attribution Techniques for Mitigating Hallucinated Information in RAG Systems: A Survey
- DRAGIN: Dynamic Retrieval Augmented Generation based on the Real-time Information Needs of Large Language Models
- Auto-RAG: Autonomous Retrieval-Augmented Generation for Large Language Models
- How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmark
- HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
- Self-Consistency Improves Chain of Thought Reasoning in Language Models