Gating Each Hop with Independent Yes/No Evidence Checks

Systems that answer multi-step questions by searching a document collection usually work in a loop. The language model reads a pile of retrieved passages, writes a reasoning step, and uses that step to search again. The weak point is the reading. At every hop the model sees many passages at once, several of them distractors that share words with the question, and nothing stops it from building the next step on the wrong one. One bad hop then poisons every search after it. This proposal replaces that joint reading with a set of small, separate decisions. For each retrieved passage, the model answers one yes/no question: "Does this passage, on its own, state the fact this hop needs?" The chain may only move forward using passages that pass. When none pass, it searches again or abstains instead of guessing.

The idea comes from multi-label classification. There, binary relevance means turning "which of L labels apply?" into L separate yes/no questions. With large language models this decomposition has produced large gains when the joint version overloads the model. It also has a clean theoretical justification when the goal is getting each individual decision right rather than producing a good ranking. Multi-hop retrieval has never treated passage selection at each hop this way. The bet is that per-hop, per-passage yes/no gating mainly reduces compounding errors, the failure where one wrong hop corrupts everything downstream, and that this is visible in hop-level metrics, in robustness to keyword-matched distractors, and in detecting unanswerable questions.

Core Claim

The evidence used at each hop of an iterative retrieval chain should be chosen by independent yes/no verdicts, one per passage, each judged against an explicit condition for that hop. The model should not choose it by reading all retrieved passages together and writing a free-form reasoning step.

Three things make this different from simply putting a reranker in the loop. First, the verdict is a set decision with an "empty set" outcome, and that outcome triggers backtracking or abstention. Second, the condition being checked is hop-specific: it is the rewritten sub-question with its pronouns resolved, not the whole original question. Third, the verdict is given without a chain-of-thought rationale, which we treat as a hypothesis to test rather than an assumption. We predict three effects: higher recall of the correct passage at hops two and beyond, fewer chains that build on a wrong bridge entity (the intermediate answer, such as a person's name, that links one hop to the next), and markedly better behaviour on unanswerable questions. We predict little change on first hops, where one-shot retrieval already works.

Why this intersection, and what prior work says

The motivating evidence comes from both directions.

On the retrieval side, interleaving reasoning with search helps a lot. IRCoT alternates chain-of-thought steps with retrieval and gains 11–21 points of recall. ITER-RETGEN shows that even a flawed draft answer supplies useful search terms. CoRAG trains on sampled retrieval chains and scales test-time search, raising MuSiQue recall from 29.0% to 47.1%. CoRAG itself names compounding failure as the open problem: an early bad retrieval leads to persistent hallucination. ChainRAG finds a related failure in which decomposed sub-questions contain vague references ("this author") and second-hop recall drops. Its fix is a retrieve-answer-rewrite loop over a sentence graph. We reuse that rewriting step to produce our hop conditions. What ChainRAG does not do is ask whether any retrieved passage actually satisfies the rewritten condition before answering it.

On the classification side, Su et al. show on movie-trope detection that asking GPT-4 one yes/no question per trope beats joint multi-label prediction by 11.8 F1. They also show that chain-of-thought hurts: it invents justifications and makes the model vulnerable to injected keywords. Guo et al. report a similar benefit from per-label binary questions for subtle linguistic categories. Both are closed-book studies with no retrieval. Gao and Zhou supply the theory. Independent binary decisions are consistent with Hamming loss, meaning per-decision correctness, while no convex surrogate is consistent with ranking loss. This matters because gating is a set-selection problem judged by per-decision correctness. A ranker trained for ordering is optimizing a different target.

Two papers constrain the design. EARnest already splits multi-hop retrieval into similarity and entailment checks. It scores a Cartesian product of candidate pairs and pays a quadratic cost that its authors flag. We avoid the pairs entirely by conditioning hop k's check on hop k−1's verified answer, which keeps cost linear in passages per hop. ReSCORE trains retrievers from an LLM's soft relevance-times-consistency scores and argues that soft labels beat hard binary labels. Its claim concerns training signal for a retriever. Ours concerns inference-time decisions about which evidence to commit to. These questions are compatible, but a reviewer will see them as in tension, so we test the point directly (see below).

Research Program

We build on the iterative loop used by ReSCORE (its IQATR setup) with Llama-3.1-8B-Instruct as the reader, so that published numbers are a reference point. At each hop the system performs four steps:

  1. Write the hop condition. The model writes the next sub-question and rewrites it to name the bridge entity explicitly, following ChainRAG's rewriting.
  2. Retrieve. The system retrieves the top k passages for that condition.
  3. Check each passage separately. For each passage it asks the reader, in an isolated prompt containing only that passage and that condition, a forced yes/no question. We read the log-odds of "yes" versus "no" as a score.
  4. Gate. Passages whose score exceeds a threshold tuned on a development set form the evidence set. The sub-answer is extracted from that set alone.

If the evidence set is empty, the system reformulates the query once and retries. If it is still empty, the chain stops and the system answers "unanswerable." The isolated prompts are short and can run in parallel. Compared with IRCoT, which accumulates all passages in context, total tokens may be similar or lower, and we will measure this rather than assume it.

The analysis has three parts.

Hop-level accuracy of the gate. MuSiQue's decompositions, 2WikiMultihopQA's evidence triples, and HotpotQA's supporting facts give hop-level gold passages. This lets us score each gate decision directly, not only the final answer.

Does chain-of-thought help the verdict? We compare verdicts given with and without a written rationale. Su et al.'s paradox was found on abstract narrative concepts, whereas multi-hop relations are mostly literal, so whether it transfers is a real open question.

Adversarial distractors. Mirroring Su et al.'s keyword-injection attack, we insert distractor passages that mention the bridge entity and the relation words but state a different or no value.

Baselines

Naive reference: one-shot retrieval. The question retrieves top-k passages once, and the reader answers. This shows the size of the multi-hop gap on our exact setup. CoRAG reports 29.0% MuSiQue recall for single-step retrieval.

IRCoT (IRCoT). This is the canonical interleaved loop and the cleanest "joint reading" competitor. We run it with the same reader, retriever, corpus, k, and maximum hop count as our method.

ITER-RETGEN (ITER-RETGEN). This is the simplest loop: it appends the draft answer to the query and retrieves again, with no decomposition. We run it for two to three iterations, its reported convergence range.

ReSCORE-trained loop (ReSCORE). This is the strongest directly comparable system. It reports 23.4 EM on MuSiQue, 47.2 EM on HotpotQA, and 50.0 EM on 2WikiMultihopQA with Llama-3.1-8B-Instruct. It also reports MHR@8 of 88.0% on 2WikiMultihopQA; MHR@8 is multi-hop recall, the fraction of questions whose full set of needed passages appears in the top eight. Because our reader matches ReSCORE's, we re-run its released configuration on our hardware rather than relying only on reported figures. Our gate can also sit on top of a ReSCORE-trained retriever, which tests complementarity rather than replacement.

CoRAG (CoRAG). It reports 47.1% MuSiQue recall and more than 10 EM over baselines. It uses trained chains and test-time search, so we compare against it at matched inference token budgets and treat it as an upper reference rather than a like-for-like baseline.

Fairness. All baselines share the corpus, the retriever (unless the retriever is the variable), k, and an equal threshold- or prompt-tuning budget on the same development split.

Ablations. Four ablations isolate the key ingredient:

  1. Joint selection. The same reader is shown all k passages and asked which ones satisfy the condition. This is the exact analogue of joint multi-label prediction in Su et al.
  2. Gate without the empty-set rule. The top-scoring passage is always used, which removes abstention and backtracking.
  3. Verdict with chain-of-thought. The model writes a rationale before answering yes or no.
  4. Condition without rewriting. The original sub-question, pronouns included, is used as the condition.

What counts as success. Beating the field means improving second-and-later-hop recall and final EM over the ReSCORE loop on MuSiQue, at no more than IRCoT's token cost. The joint-selection ablation must lose to the independent gate. If it does not, the binary-relevance ingredient is not what is helping.

Why not just…?

Why not just use a trained cross-encoder reranker on the sub-question? This is the most serious threat. A strong reranker scoring each passage against the rewritten condition already makes per-passage decisions. We will include it as a direct competitor, thresholded and given the same empty-set rule. If it matches the LLM gate, the contribution reduces to "gate with an abstain rule," and the gate can use whichever scorer is cheaper. Two reasons suggest it may not match. Rankers are trained for ordering, not for calibrated set membership, which is the distinction Gao and Zhou formalize. And hop conditions in MuSiQue are compositional rewrites that differ from typical reranker training queries. The experiment will settle this, and we will report it either way.

Why not just let the model generate free-form sub-questions and answer them, as in ChainRAG or CoRAG? These methods fix what to search for but still commit to whatever the reader extracts from the retrieved pile. Our ablation keeps ChainRAG-style rewriting and removes only the gate. That comparison measures how much error comes from accepting unsupported evidence rather than from vague queries.

Why not just run ITER-RETGEN, which needs no decomposition at all? Its bridge terms come from the model's own draft. When the draft names the wrong entity, the next retrieval is aimed at the wrong target, and on MuSiQue's connectivity-filtered questions surface-similarity shortcuts lose about 30 F1 (MuSiQue). It remains a plausible competitor on HotpotQA, where shortcuts work, and we expect our advantage to be concentrated on MuSiQue.

Why not just train the retriever better with soft labels, as ReSCORE argues? Better retrieval and better gating address different stages, so the honest test is to stack them. If the ReSCORE retriever plus our gate does not beat the ReSCORE retriever alone, then the gate adds nothing once retrieval is strong. We will also test the narrower point of tension: whether a retriever fine-tuned on hard hop-wise gate verdicts approaches one fine-tuned on soft ReSCORE scores.

Proposed Experiments

The experiments run on MuSiQue-Ans, HotpotQA (distractor and full-wiki settings), and 2WikiMultihopQA. We report answer EM and F1, recall split by hop index, gate precision and recall against hop-level gold passages, the fraction of chains that commit to a wrong bridge entity, and inference tokens. We expect HotpotQA gains to be smaller, since bridge questions make up only 42% of it (HotpotQA).

First decisive check. Gains should come almost entirely from hops two and beyond, and the wrong-bridge commitment rate should fall. If first-hop recall improves just as much, the method is working as a generic reranker, not as a guard against compounding errors.

Second decisive check. Joint selection must underperform independent verdicts at an equal number of candidate passages.

Third decisive check: unanswerable questions. On MuSiQue-Full, which pairs answerable questions with unanswerable contrast versions, the empty-set rule should give a principled unanswerable prediction. Lacuna has no reported result for answerability detection by iterative RAG systems on MuSiQue-Full, so we will establish the IRCoT and ReSCORE-loop numbers ourselves.

Fourth decisive check: chain-of-thought and distractors. We test whether written rationales help or hurt verdicts, overall and under injected keyword-matched distractors. If verdicts with rationales are as robust as direct verdicts on literal hops, we will report that Su et al.'s paradox does not transfer to factual hops. That outcome would be a useful constraint on when binary decomposition should avoid chain-of-thought.