Multi-hop questions such as "In what city was the director of Inception born?" need a chain of documents, and the documents in the middle of that chain are never named in the question. Usually nobody has annotated which documents those are, so a system that learns to find them is training a latent variable model: the hidden variable is the chain, and only the question and the answer are observed. Training such a model properly means maximizing the probability of the answer averaged over all chains. That sum is intractable over a five-million-article corpus. Current systems avoid the sum in one of two ways. Some demand gold intermediate annotations. Others sample chains from the current retriever and keep only the ones that reach the answer. This second approach breaks at three and four hops, because almost no chain sampled forward from the question gets that far.
We propose to train the multi-hop retriever by importance-weighted marginal likelihood. The key ingredient is a training-time "posterior" retriever that can see the answer and proposes chains working back toward it. Each proposed chain is scored by how plausible it is under the deployable question-only retriever and by how well it lets a frozen reader produce the answer. The question-only retriever is then trained on these soft-weighted chains. A persistent per-question buffer of good chains is carried across epochs. The prediction is specific: as hop depth grows, this method should pull steadily ahead of rejection-sampling training at equal sampling cost, because the answer-aware proposal keeps the fraction of useful samples from collapsing.
Core Claim
Write $x$ for the question, $y$ for the answer, and $z=(d_1,\dots,d_T)$ for a chain of retrieved documents ending in a stop action. Let $p_\theta(z\mid x)$ be the deployable retriever, which produces each next-hop query from the question and the documents retrieved so far. Let $p(y\mid x,z)$ be a frozen reader language model. Let $q_\phi(z\mid x,y)$ be an answer-aware proposal retriever with the same architecture plus the answer string as input. With $K$ chains $z_k$ drawn from $q_\phi$ we optimize
$$\log p_\theta(y\mid x)\;\ge\;\mathbb{E}\Big[\log \frac{1}{K}\sum_{k=1}^{K} w_k\Big],\qquad w_k=\frac{p_\theta(z_k\mid x)\,p(y\mid x,z_k)}{q_\phi(z_k\mid x,y)}.$$
This is the multi-sample bound used for hierarchical posteriors in Importance Weighted Hierarchical Variational Inference, which tightens evidence lower bounds by importance weighting. Here it is applied to a discrete chain of documents. The retriever $p_\theta$ is trained with the normalized weights $\bar w_k$ as targets for its log-probabilities. The proposal $q_\phi$ is trained toward the same weighted chains, so it tracks the posterior over chains. Neither model ever sees a gold supporting fact.
The claim we test has two parts. First, question-answer pairs alone suffice to train a retriever that is competitive with annotation-supervised ones. Second, the benefit over filter-by-correctness training grows with hop count. That growth is the falsifiable part.
Relation to Prior Work
Four papers sit on this exact boundary, and each motivates or constrains the design.
Hop, Union, Generate treats supporting sentences as a latent set and approximates the marginal likelihood with top-$K$ Viterbi search. It reaches 67.1 sentence F1 without rationale labels on HotpotQA distractor. However, it trains 26x slower and is confined to ten pre-filtered paragraphs, because it enumerates candidates. We sample instead of enumerate, so we can work over the open corpus. We also average over chains with soft weights rather than keeping only the highest-scoring ones.
Beyond Verifiable Rewards supplies the training principle: treat the chain of thought as latent and use the answer's log-likelihood as the learning signal, with more samples tightening the bound. It has no retrieval, and the chains it samples come only from the forward model. We move the latent variable from text to documents and add a separate answer-conditioned proposal.
Momentum Posterior Regularization is the closest relative. It distills an answer-aware "posterior retriever" into a question-only one. It reports that naive posterior regularization is unstable in multi-hop settings, and it fixes this with LLM summaries of the gold hops and a moving-average teacher. That gold dependence is exactly what we remove. In our setup the importance weights, not the gold summaries, determine where the proposal should put mass. MoPo's instability finding constrains our design: we mix in prior samples and use the buffer specifically to keep $q_\phi$ from running ahead of $p_\theta$.
Chain-of-Retrieval Augmented Generation trains on retrieval chains that reach the correct answer, found by rejection sampling. This is the hard, forward-sampled special case of our objective, and it is the method we must beat.
From the marginal-likelihood literature, two papers supply the buffer idea. SOUL shows that warm-starting inexpensive, inexact samplers across optimization steps makes stochastic-approximation EM practical in high dimensions. Stochastic Approximation with Biased MCMC for Expectation Maximization shows that accept/reject samplers stall when acceptance rates collapse, while biased samplers without that step keep making progress. We draw an analogy and do not claim the theory transfers: rejection sampling over four-hop chains is an accept/reject scheme with near-zero acceptance. A reweighted persistent buffer is the biased-but-moving alternative.
We deliberately do not use the continuous Langevin and particle methods (Momentum Particle Maximum Likelihood, Momentum SVGD-EM). They need gradients with respect to the latent variable, and a chain of retrieved documents has none. Inventing a continuous relaxation would add complexity without addressing the actual bottleneck, which is proposal quality.
Research Program
The retriever $p_\theta$ is a dense query encoder. At each hop it reads the question and the retrieved passages and outputs a query vector. A stop action ends the chain. Hop probabilities are a softmax over the top-$M$ passages that maximum-inner-product search returns from the index. This truncation introduces a known bias, and we measure it by varying $M$.
The proposal $q_\phi$ shares this architecture and additionally takes the answer string as input. Because it knows where the chain must end, it can retrieve a document containing the answer and work back toward the bridge entities. Optionally it is given an alias set for the answer. In that case the reader likelihood sums over aliases, which partly addresses the exact-string rigidity noted for JEPO.
The reader is a frozen 8B instruction model. It scores $\log p(y\mid x,z)$ in a single forward pass per chain, so the reader is never used to generate during training.
Each training step works as follows for a question. We draw $K/2$ chains from $q_\phi$ and $K/2$ from $p_\theta$. We add the buffer's chains for that question and rescore all of them under the current models. We update both retrievers with the normalized weights. Finally, we keep the top chains by weight in the buffer.
We will analyze three diagnostics as functions of hop count (MuSiQue's 2/3/4-hop split): the effective sample size of the weights, the fraction of questions with at least one answer-reaching chain, and how often the highest-weighted chain matches gold supporting facts. Gold supporting facts are used for evaluation only. At test time the system is the question-only retriever followed by one reader call. Its cost is therefore comparable to L-RAG and EfficientRAG, not to tree search.
Baselines
The naive reference point is single-step dense retrieval followed by the same reader. CoRAG reports MuSiQue retrieval recall of 29.0% for this setting.
The strongest competitor on the precise question we ask is CoRAG-style rejection-sampling training. CoRAG reports MuSiQue recall rising to 47.1% with its trained chains plus test-time search. We will reimplement its training signal on our architecture: sample chains from $p_\theta$, keep those whose reader output matches the answer, and fine-tune on them. It gets the same reader, index, encoder, and total number of sampled chains and reader scoring calls per question, plus an equal hyperparameter search budget. Beating it means higher recall@k and answer accuracy on the 3- and 4-hop MuSiQue subsets at matched training compute, with confidence intervals over three seeds that do not overlap. Parity on 2-hop is acceptable.
The supervised reference is MoPo. It reports HotpotQA EM@100 of 82.20 against 79.70 for MDR, using gold-derived summaries. We do not expect to beat it. Closing most of the MDR-to-MoPo gap without any annotations would count as success.
For end-to-end efficiency, the reference is L-RAG, which reports 0.352 MuSiQue accuracy at about 2.8 seconds per query on 2WikiMultiHopQA. IRCoT, which reports up to 15 F1 points over one-step retrieval, will be run with the same reader as the iterative-prompting reference.
Three ablations isolate the ingredients. The first replaces $q_\phi$ with $p_\theta$, giving soft weights but forward proposals; this separates the effect of the answer-aware proposal from that of soft weighting. The second hardens the weights to a correct/incorrect filter. The third removes the buffer.
Why not just…?
Why not just do rejection sampling with more samples, as CoRAG does? This is the real threat. If the fraction of answer-reaching chains sampled forward stays reasonable on 4-hop questions, extra samples may close the gap. The hop-stratified acceptance-rate measurement settles this directly. We will plot rejection-training accuracy against sample budget up to 4x ours. If it matches our method at 4 hops within that range, the answer-aware proposal is unnecessary, and the contribution reduces to soft weighting. The first ablation tests that remaining contribution separately.
Why not just generate a backward summary from the answer with an LLM and plug it into MoPo? This is a cheap way to drop the gold dependence. The summary, however, would be the LLM's guess about the bridge facts, with nothing tying it to documents that actually exist in the corpus. We include this as a baseline: MoPo with answer-only LLM summaries. Our expectation is that it works on 2-hop bridge questions and degrades where parametric knowledge of the intermediate entities is weak. We will check that by stratifying results by entity popularity.
Why not just spend the compute at test time with tree search, as GRAT does? GRAT reaches 75.2% on 2WikiMultiHopQA, but with a 70B model and rollouts at every node. Test-time search is complementary to our method, not a substitute for it. We will report accuracy against inference tokens, and we will also run best-of-N search on top of our trained retriever. If search on the CoRAG-trained retriever dominates our curve at every budget, our training offers no practical advantage.
Proposed Experiments
The experiments use HotpotQA fullwiki, 2WikiMultiHopQA, and MuSiQue with open corpora in the IRCoT-style setup. StrategyQA is held out for zero-shot transfer and compared against MoPo's reported 39.24 EM@100 versus MDR's 36.06. HotpotQA distractor is used to compare supporting-sentence F1 against HUG's reported 67.1. All training uses question-answer pairs only. Metrics are retrieval recall@k, answer exact match and F1, supporting-fact F1 (evaluation only), latency per query, and training reader calls.
The decisive check is the hop-depth curve: the gap between our method and rejection training at matched compute, plotted against hop count on MuSiQue. The hypothesis predicts a gap that grows with depth, along with a higher effective sample size and answer-reaching fraction at 4 hops. A flat or shrinking gap falsifies the core claim.
Secondary checks address the remaining risks. The effective sample size over training should rise, which would show that $q_\phi$ is learning a useful posterior. Varying $M$ shows how much the truncated softmax biases the weights. Finally, examining high-weight chains that do not use gold supporting facts tests whether the reader is being rewarded for shortcuts. On 2WikiMultiHopQA, which was built to resist such shortcuts, this rate should be low.
References
- Importance Weighted Hierarchical Variational Inference
- Hop, Union, Generate: Explainable Multi-hop Reasoning without Rationale Supervision
- Beyond Verifiable Rewards: Scaling Reinforcement Learning in Language Models to Unverifiable Data
- Momentum Posterior Regularization for Multi-hop Dense Retrieval
- Chain-of-Retrieval Augmented Generation
- Efficient stochastic optimisation by unadjusted Langevin Monte Carlo
- Stochastic Approximation with Biased MCMC for Expectation Maximization
- Momentum Particle Maximum Likelihood
- Momentum SVGD-EM for Accelerated Maximum Marginal Likelihood Estimation
- Optimizing Multi-Hop Document Retrieval Through Intermediate Representations
- GRAT: Guiding Retrieval-Augmented Reasoning through Process Rewards Tree Search