Meeting in the Middle: Training Deep Multi-Hop Retrievers from Answers Alone

Some questions need several documents chained together. "In what city was the director of Inception born?" needs one passage to name the director and a second to give his birthplace. A search system for such questions has to learn which document to fetch at each step. The training data almost never says which intermediate documents were the right ones. It usually gives only the question and the final answer. Trained on that signal alone, a retriever is rewarded whenever the answer comes out right, even when the chain of documents behind it is nonsense. That happens often: on a counterfactual benchmark, GPT-4 produced fully correct reasoning chains only 36.3% of the time while still frequently getting final answers right (CofCA). The problem gets worse with each added hop, because the number of possible chains grows and lucky guesses become more common.

We propose to search from both ends of the chain during training. One retriever walks forward from the question. A second walks backward from the known answer, starting at passages that contain the answer and asking which passage could have led there. On the training set, a chain is trusted only where the two walks agree on the same middle passage. Those agreement points become synthetic labels for the intermediate hops, and a forward-only retriever is trained on them for use at test time, when no answer is available. The idea comes from knowledge-graph question answering, where forward and backward walks over graph edges were reconciled to suppress spurious paths (He et al.). Nobody has carried it over to retrieval over unstructured text, where there are no invertible edges. Our claim is that this agreement constraint gives cleaner intermediate supervision than filtering trajectories by final-answer correctness, and that the advantage grows with chain depth, specifically at three and four hops.

What Each Line of Work Gives Us, and What It Leaves Open

The three directions play different roles here. Weak intermediate supervision is the core problem we attack. Multi-hop retrieval-augmented generation is the setting: a system that fetches passages from a corpus and passes them to a language model that writes the answer. Scaling beyond two hops is the stress test that separates our method from simpler ones.

Three papers supply ingredients. He et al. train a teacher network that spreads probability forward from the question entity and backward from the answer entity over a knowledge graph. It minimizes the divergence between forward step $k$ and backward step $n-k$, and uses the result to supervise a student. That raised WebQSP Hits@1 from 68.7% to 74.3%. Their method needs graph edges and a fixed hop count. We need neither, and we replace graph traversal with dense passage retrieval. Baleen shows that hop order can be treated as latent by picking whichever gold document is easiest to find next. However, its condenser still needs sentence-level gold labels. Q-RAG trains only the embedding model for multi-step retrieval with value-based reinforcement learning, cheaply, on a single A100 in about 12 hours. Its authors state that it requires gold support-fact labels to compute rewards and that answer-only training remains open. We reuse Q-RAG's choice to reason in embedding space with a frozen generator, and we aim to remove its dependence on gold labels.

Two lines show that answer-only training is possible but limited in scale. Hop, Union, Generate (HUG) selects documents and sentences as hidden variables trained purely from answers. It reaches 67.1 sentence F1 on HotpotQA in the distractor setting, a small pool of ten paragraphs per question. But it is 26x slower to train, and it cannot scale to an open corpus. Chain-of-Questions treats sub-answers as hidden variables trained with Hard-EM and reinforcement learning, and finds that this structure resists shortcuts. It still needs sub-question decompositions and does not do dense open-domain retrieval.

Tree-search systems are the strongest current answer to unlabeled intermediate steps. GRAT and DecoupleSearch run Monte Carlo tree search, a search that expands many candidate reasoning paths and scores them by simulated outcomes, and keep or score paths by their outcomes. Their supervision ultimately rests on final-answer correctness or an LLM judge. Both papers note high search cost. Neither checks an intermediate step against independent evidence from the answer side.

Two other papers constrain the design and motivate the depth focus. Hopping Too Late and Back Attention show that models resolving a chain inside their own layers run out of depth as hops accumulate. Methods that read a retrieval query from one fixed middle layer, such as L-RAG, are built and tested mainly for two hops. We therefore make each hop an explicit retrieval step, and we do not rely on depth inside the network.

Core Claim

Requiring a forward walk from the question and a backward walk from the answer to land on the same intermediate passage yields intermediate labels whose precision degrades much more slowly with hop count than labels from final-answer filtering. Our reasoning is about independence. A spurious forward chain reaches the right answer through parametric guessing or a shortcut. The backward walk is anchored on answer-bearing text and never sees the forward chain's mistakes, so it rarely meets that chain by coincidence. Final-answer filtering has no such check. Its false-positive rate should grow with depth, because there are more places to go wrong while still guessing correctly.

Research Program

The system has a forward query encoder and a backward query encoder, both dense bi-encoders over a shared passage index. The forward state at hop $k$ encodes the question together with the passages retrieved so far. The backward state at hop $j$ encodes the question, the answer string, and the passages retrieved backward so far.

The first backward hop is anchored by restricting candidates to passages that contain the answer string. This is the only place the answer enters. Each encoder induces a softmax distribution over a candidate pool. The pool is the union of top-$m$ results from both directions, so the two distributions share support.

For an assumed chain length $n$, the teacher's agreement loss is

$$ L_{\text{agree}}(n) = \sum_{k=1}^{n-1} \mathrm{JS}\big(p^{\text{fwd}}_k \,\|\, p^{\text{bwd}}_{n-k}\big), $$

where $p^{\text{fwd}}_k$ is the forward distribution over candidates at hop $k$, $p^{\text{bwd}}_{n-k}$ is the backward distribution at the matching backward hop, and JS is Jensen-Shannon divergence. The two endpoints are also constrained. The forward final hop must reach an answer-bearing passage, and the backward final hop must reach a passage sharing an entity with the question. Because the hop count is unknown, we marginalize over $n \in \{2,3,4\}$ with a softmax over $-L_{\text{agree}}(n)$. This lets the data choose the depth.

Training alternates in an EM-like loop. In the first step, the teacher's forward-backward agreement produces soft labels over intermediate passages. In the second step, both encoders are updated toward those labels, using in-batch negatives. A forward-only student, which also predicts when to stop, is distilled from the teacher's agreed chains. Only the student runs at test time. Its cost is one encoder call per hop and one call to a frozen reader model, with no text generation between hops.

The main risk is collapse: both directions could agree on a generic, high-traffic passage. Three things guard against it. The endpoint anchors tie the chain to the question and the answer. Every candidate pool includes hard negatives. And we monitor whether agreed passages are concentrated on a few "hub" passages. If collapse appears anyway, we will report it as a finding, not tune it away silently.

The data covers bridge-style questions whose answers are entity strings: MuSiQue-Ans at 2 to 4 hops, 2WikiMultiHopQA, the bridge subset of HotpotQA, and CofCA at 2 to 4 hops. Yes/no questions, comparison questions, and claim verification such as HoVer have no answer string to anchor the backward walk, so they are out of scope. We say this up front.

Gold supporting paragraphs exist in MuSiQue and HotpotQA. We never use them in training. We use them only to score label precision and retrieval recall. This lets us measure the quality of the synthetic intermediate labels directly, not only through downstream accuracy.

Baselines

The naive reference point is single-step dense retrieval with an untrained Contriever-style encoder, followed by the same frozen reader. This is the one-shot setting the multi-hop gap is defined against. On HotpotQA the file gives no matching dense number. The original BiDAF full-wiki baseline reached only 17.73 joint F1 (HotpotQA), which shows how low the floor is.

The strongest competitor on the actual question, answer-only training of intermediate steps, is outcome-filtered self-training in the style of GRAT. We sample forward retrieval chains, keep those whose final passage yields the correct answer, and train the same forward encoder on the kept chains. GRAT reports that this kind of self-training raised Llama3-8B on 2WikiMultiHopQA from 60.6% to 66.8% accuracy. We run it with the identical encoder, corpus, reader, number of sampled chains per question, and GPU-hour budget, so the only difference is the filtering rule. Beating it means higher label precision against gold MuSiQue paragraphs, higher per-hop recall, and higher answer F1. We expect most of the gain on 3- and 4-hop questions.

Q-RAG is the strongest embedding-space method (Q-RAG). We run it in two forms. One uses its original gold support-fact reward, serving as a supervised ceiling. The other uses a terminal-only reward for reaching an answer-bearing passage, which is the direct answer-only competitor. The file reports no MuSiQue or HotpotQA number for Q-RAG, so we must reproduce both versions ourselves under the same budget.

For context on end-to-end systems we report IRCoT and TSSS. TSSS reports 34.1 EM on HotpotQA against 28.0 for IRCoT, and 9.5 seconds per sample on MuSiQue against 27.2 for Iter-RetGen (TSSS). We also report L-RAG, which reports 0.352 accuracy on MuSiQue at about 2.8 seconds per query on 2WikiMultiHopQA (L-RAG). In the ten-paragraph distractor setting we compare against HUG's 67.1 sentence F1.

Three ablations isolate the key ingredient. The first removes the backward walk and the agreement loss, keeping everything else. This is close to the outcome-filtered baseline, but with soft labels. The second keeps the backward walk but uses its retrievals as separate labels, without requiring agreement. This tests whether meeting in the middle matters or whether answer-side retrieval alone helps. The third fixes $n$ at the gold hop count instead of marginalizing over it.

Why not just…?

Why not just filter self-generated chains by final-answer correctness? This is the real threat, and the experiments are built around it. CofCA's finding that correct answers often come with broken chains suggests that answer filtering admits many spurious intermediate passages. MuSiQue's 30-point F1 drop for shortcut models compared with HotpotQA (MuSiQue) shows that shortcuts are common on weaker datasets. Still, if outcome filtering matches our label precision at 3 and 4 hops, the proposal fails. The label-precision measurement against gold paragraphs settles this directly.

Why not just run Q-RAG with a terminal reward and let value bootstrapping assign credit to intermediate hops? Bootstrapping can in principle spread sparse reward backward. With a reward based only on answer-bearing passages, though, every chain that stumbles onto such a passage gets credit, which is the same spurious-path problem in a different form. We run exactly this variant. If it matches us, the conclusion is that the backward walk is unnecessary.

Why not just label intermediate steps with an LLM judge or tree search, as DecoupleSearch does? That is a reasonable but costly alternative: DecoupleSearch itself flags the cost of tree-search annotation. A judge also shares the same parametric biases that produce shortcut answers. We include a judge-labeling variant at a matched annotation budget on a subset, and we compare label precision against gold.

Why not just prompt a strong model to iterate retrieval, as IRCoT does? It works, but it pays a large latency cost per hop, and its linear chain cannot backtrack. The latency numbers above make this a deployment question as much as an accuracy question. We report both accuracy and seconds per query.

Decisive Checks

Results are stratified by hop count on MuSiQue and CofCA. The core prediction is that the label-precision gap between agreement-based and answer-filtered supervision is small at two hops and widens at three and four. A flat gap would mean the depth argument is wrong.

CofCA's counterfactual split is the cleanest test of the independence argument. When the reader's parametric memory points to the wrong answer, spurious forward chains should fail the agreement check more often than they fail the answer filter. We report CofCA's Joint Performance metric, which scores intermediate and final correctness together.

Finally, the marginal over $n$ should put its mass on the gold hop count well above chance, measured against MuSiQue's hop labels. If it does not, the method is not actually recovering chain structure, whatever its downstream accuracy.