Questions like "In what city was the director of Inception born?" can't be answered with a single search. A language model has to search once, find the bridge entity (Christopher Nolan), and then issue a second search that names that entity. Current trained systems learn to write these intermediate searches by imitating successful search chains. Imitation of this kind is "mean-seeking": the model is pushed to give some probability to every successful variant it saw. As a result, at each hop it spreads probability over many different next queries. Some are sharp ("Christopher Nolan birthplace"). Others are vague ("where was this director born") and retrieve nothing useful. Once a vague query is sampled, the chain drifts, and the error compounds.
We propose to train the query-writing step with a "mode-seeking" objective instead. Such an objective penalizes the model for putting probability on low-value options, so its mass concentrates on the few queries that actually retrieve the missing evidence. Two design choices separate this from simply swapping one loss for another. First, every comparison is made between alternative next queries that branch from the same shared history, so the conditioning context is identical across compared candidates. Second, each candidate's reward is checked automatically: did the query retrieve a gold supporting paragraph, and do continuations from it reach the right answer? We also define a measurable quantity, the spread of the policy's next-query distribution over groups of queries that retrieve the same documents. This tests directly whether chains wander because probability is diluted across query types.
Core Claim
Part of the multi-hop reasoning gap comes from how probability is spread at each hop, not only from what the model knows. A policy trained by imitation on correct chains spreads its next-query probability across many retrieval outcomes. We call each distinct outcome a "retrieval mode": a set of sub-queries that return the same top-k documents. Wandering and redundant search are samples from the low-value modes.
Mode-seeking alignment applied hop by hop can concentrate mass on gold-retrieving modes. This should raise later-hop recall and final exact match while shortening chains. It should also beat both the cheap sharpening tricks (temperature, picking one canonical chain) and standard Direct Preference Optimization (DPO, which trains on chosen-versus-rejected response pairs) on identical data.
Relation to Prior Work
Work that motivates the proposal. Two papers document the failure we target. ChainRAG shows that sub-queries with unresolved references ("this author") cut second-hop Recall@2 by up to 18%. It repairs these with a rewriting loop driven by prompting rather than training. EVO-RAG names the redundancy, shallow-exploration and wandering failures and trains sub-query generation with standard DPO over a seven-factor reward and a curriculum. We keep its sequential-decision view but drop the hand-weighted reward vector, change the loss, and change how pairs are built.
The strongest trained system, and our starting point. CoRAG samples chains of sub-queries and sub-answers, keeps those that reach the correct answer, and fine-tunes Llama-3.1-8B-Instruct by standard cross-entropy. It raises MuSiQue retriever recall from 29.0% to 47.1%. We use its checkpoint recipe as our reference policy. Our question is whether a mode-seeking second stage helps on top of it.
Work that constrains the design. RPO proves that DPO's derivation breaks when the retrieved context differs between the chosen and rejected response. The normalizing terms no longer cancel. Whole-trajectory preference pairs in multi-hop retrieval violate exactly this condition, because different queries retrieve different documents downstream. That is why every comparison in our proposal shares an identical prefix.
The literature also disagrees about what DPO actually optimizes. H-DPO treats it as reverse-KL-regularized but too high in entropy. EXO and GraphMPA describe it as behaving like mean-seeking forward KL. Our factorial design separates these explanations empirically rather than taking a side.
EXO also warns that mode-seeking objectives exploit noisy reward models. We therefore use no learned reward model, only retrieval and answer checks against gold annotations.
Work that supplies the method. EXO directly minimizes reverse KL between the policy and a reward-tilted target. It is our primary loss. InfoPO and SimPER are alternative mode-seeking losses. Both report large GSM8K gains over DPO, for example 49.13 versus 38.67 for InfoPO on Llama-3-8B. They serve as variants.
The closest existing work. GraphMPA already pairs reverse-KL alignment with retrieval. However, it applies the loss only to final answer generation over static community summaries, and it never evaluates on HotpotQA, MuSiQue or 2WikiMultiHopQA. We apply the loss to the intermediate query decisions and evaluate on exactly those benchmarks.
Research Program
Building the training data
We start from a CoRAG-style policy $\pi_{\text{ref}}$ trained on the HotpotQA and MuSiQue training sets. Both datasets annotate supporting paragraphs. We roll out chains and record each "state" $s$: the question, the previous sub-queries, the previous sub-answers and the retrieved documents. From each state we sample $K=8$ candidate next sub-queries $q_1,\dots,q_K$ with a fixed retriever.
Each candidate receives a verifiable reward $r(s,q_i)$. The reward adds two parts. The first is an indicator that the query retrieves a not-yet-seen gold supporting paragraph in its top-$k$. The second is the fraction of $M=4$ continuations from that query that end at the correct answer.
The loss
All candidates share the same state $s$, so the RPO objection does not apply. Following EXO, we define the policy's normalized preference over the candidate set as
$$p_\theta(i\mid s)=\frac{\exp\big(\beta\log\frac{\pi_\theta(q_i\mid s)}{\pi_{\text{ref}}(q_i\mid s)}\big)}{\sum_j \exp\big(\beta\log\frac{\pi_\theta(q_j\mid s)}{\pi_{\text{ref}}(q_j\mid s)}\big)}.$$
Here $\pi_\theta$ is the policy being trained and $\beta$ controls how far it may move from $\pi_{\text{ref}}$. The target distribution is $p_r(i\mid s)\propto\exp(r(s,q_i)/\tau)$, with temperature $\tau$. We minimize
$$\mathcal{L}(\theta)=\mathbb{E}_s\,D_{\mathrm{KL}}\big(p_\theta(\cdot\mid s)\,\|\,p_r(\cdot\mid s)\big).$$
This reverse direction heavily penalizes any mass the policy places on candidates whose reward is near zero. It is cheaper than whole-chain optimization: each hop decision is a short sub-query, and all candidates are scored under one shared context.
The diagnostic
For any state, we group sampled sub-queries into retrieval modes by their top-$k$ document set. We then report two quantities: the entropy over modes, and the mass on gold-retrieving modes.
The mean-seeking hypothesis makes a falsifiable prediction. Under the imitation-trained policy, mode entropy at hops two and later should be high. It should also predict, at the question level, whether a chain wanders (repeats queries or exceeds the gold hop count) or fails. Alignment should then lower mode entropy specifically by removing non-gold modes, not by uniformly sharpening everything.
Baselines
Naive and prompt-only references. The reference point for doing nothing beyond the starting policy is the CoRAG-style imitation policy with greedy decoding. The prompt-only reference is IRCoT, which alternates chain-of-thought sentences with retrieval and needs no training. Both run with the same retriever and corpus.
Strongest existing trained method. This is CoRAG, reproduced at 8B on our retriever. Its reported numbers are MuSiQue retriever recall of 47.1% (up from 29.0% single-step) and an advantage of more than 10 EM points over IterDRAG and Search-o1-32B on 2WikiQA and MuSiQue. No reported result found on Lacuna for CoRAG's absolute EM on these benchmarks, so we report our reproduced numbers. Beating it means two things at equal training rollouts and equal test-time calls. First, higher MuSiQue and 2WikiMultiHopQA EM, with a gain larger than the across-seed spread over three seeds. Second, an equal or shorter average chain.
Strongest preference-based competitor. This is EVO-RAG's DPO training. It reports up to +4.6 EM and 15% shallower retrieval depth. We reimplement it twice. One version uses its trajectory-level pairs. The other applies DPO to our own shared-prefix pairs, built from the best and worst candidate per state under the same rewards. Both versions get the same tuning budget for $\beta$ that our method gets.
For context against dynamic retrieval, DRAGIN reports 2WikiMultihopQA F1 of 0.3931 with LLaMA2-13B. That model differs from ours, so the number is context only.
Ablations isolating the key ingredient. We run a two-by-two design. One axis is the pair construction: whole trajectories versus shared-prefix hops. The other axis is the loss: DPO versus EXO-style reverse KL. InfoPO and SimPER losses on shared-prefix data serve as further mode-seeking variants. This design attributes any gain to the context-matching fix (RPO's account), the divergence (EXO's account), or both. A further ablation removes the continuation-success term from the reward, which tests whether retrieval hits alone are enough.
Why not just…?
Why not just lower the temperature or decode greedily? If wandering were only excess randomness, sharpening at decode time would fix it for free. H-DPO reports the relevant comparison: at matched entropy, mode-seeking training beats temperature-sharpened DPO. That evidence comes from GSM8K, not retrieval. Temperature sharpens whatever mode is locally largest, including a vague-referent query that is common in the training chains. We test this directly. Temperature-swept versions of the reference policy are compared at matched mode entropy. If they reach the aligned policy's gold-mode mass, our claim fails.
Why not just fine-tune on the single shortest correct chain per question? This is the most serious threat. Keeping one canonical chain makes imitation data unimodal, which approximates mode-seeking without any new loss. We include it as a full baseline with the same rollout budget. It differs from our method in one respect: it never sees negative candidates at a specific hop, so it cannot learn to avoid vague queries. If it matches our method, the honest conclusion is that data selection, not the divergence, does the work, and we would report that.
Why not just spend the compute at test time with Best-of-N or tree search? CoRAG shows that test-time search scales log-linearly. We compare at matched total retriever and model calls. The aligned policy uses greedy decoding plus the leftover budget. CoRAG uses Best-of-N. If search wins at every budget, training-time sharpening has no value beyond reducing cost.
Why not just use DPO with step rewards, as EVO-RAG does? The shared-prefix DPO cell of our ablation is exactly this comparison, with the context problem removed. If it equals the reverse-KL cell, the benefit comes from fixing the context mismatch rather than from mode-seeking. That would still be a useful, cheaper finding.
Proposed Experiments
Setup. We train on the HotpotQA and MuSiQue training splits. We evaluate in-domain on HotpotQA (Full Wiki setting) and MuSiQue-Ans. Out of distribution, we evaluate on 2WikiMultiHopQA and Bamboogle, the latter built adversarially to defeat single-query search. The model is Llama-3.1-8B-Instruct with one fixed dense retriever over the Wikipedia corpus for every method.
Metrics. We report answer EM and F1, gold-paragraph recall at each hop, average number of hops, and redundancy rate (repeated or near-duplicate queries). We also report the vague-referent rate: the share of queries containing an unresolved reference, which an automatic check flags. Finally, we report the mode entropy and gold-mode mass defined above.
Decisive checks. There are four.
- Reverse-KL alignment on shared-prefix data must beat shortest-correct-chain imitation and shared-prefix DPO on MuSiQue EM by more than seed variance. If it does not, the mode-seeking story is unsupported.
- The gain must concentrate at hops two and later, matching ChainRAG's locus of second-hop recall loss. It should also come with a drop in vague-referent rate. Uniform gains across hops would point to a generic training effect.
- At the question level, mode entropy under the reference policy must predict wandering and failure better than token-level entropy, which is DRAGIN's trigger signal. If token entropy predicts equally well, the retrieval-mode construct adds nothing.
- On Bamboogle and 2WikiMultiHopQA, the aligned policy must not lose to the reference. A loss there would indicate the sharper policy overfit to HotpotQA and MuSiQue query patterns, the generalization risk EVO-RAG's single-dataset training also faces.
References
- Mitigating Lost-in-Retrieval Problems in Retrieval Augmented Multi-Hop Question Answering
- Curriculum Guided Reinforcement Learning for Efficient Multi Hop Retrieval Augmented Generation
- Chain-of-Retrieval Augmented Generation
- RPO: Retrieval Preference Optimization for Robust Retrieval-Augmented Generation
- Entropy Controllable Direct Preference Optimization
- Towards Efficient Exact Optimization of Language Model Alignment
- A Comprehensive Graph Framework for Question Answering with Mode-Seeking Preference Alignment
- InfoPO: On Mutual Information Maximization for Large Language Model Alignment
- SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
- DRAGIN: Dynamic Retrieval Augmented Generation based on the Real-time Information Needs of Large Language Models