When a language model answers a question like "In what city was the director of Inception born?" by alternating between writing a reasoning sentence and searching a document collection, one bad search early on usually dooms the answer. Current ways of training such agents with preference learning have two weaknesses. They judge whole trajectories at once, so a good first hop is penalized along with a bad third hop. They also assume every training question comes with a verified gold answer. This proposal trains the agent on single-hop preference pairs. We branch the agent's own trajectory at each hop, resume it several times from alternative next actions, and prefer the action whose resumptions more often succeed. We also ask how few gold answers this recipe really needs, which bears directly on the cost of human feedback.
The method rests on Direct Preference Optimization (DPO), which trains a model directly on "this output is better than that one" pairs, with no separate reward model. The setting is interleaved retrieval-augmented generation, where the model's reasoning steps become search queries. The human-feedback bottleneck enters as a constraint: we train with gold labels on only a small fraction of questions, chosen actively, and we measure what is lost.
Core Claim
Our first claim is that, for multi-hop retrieval agents, hop-level pairs built from the agent's own branched rollouts beat three alternatives on multi-hop QA accuracy at equal compute. Those alternatives are trajectory-level DPO, fine-tuning on successful trajectories, and prompt-only interleaving. We use the term "hop" for one reasoning sentence together with the search query it issues.
The second claim is that the pair construction needs gold answers for only a small, actively chosen subset of questions. For the rest, agreement among resumed rollouts can stand in for correctness, provided that a small labeled seed is used to calibrate when agreement can be trusted.
Both claims are falsifiable on existing benchmarks. The second is the part that connects to the feedback bottleneck, and it is the part most likely to fail.
Relation to Prior Work
Several works motivate this design by showing that step-level contrast matters, but they all live in closed-world reasoning. Step-DPO contrasts the first wrong step of a math solution against a corrected one. It shows that self-generated errors train better than errors produced by GPT-4, and it relies on programmatic gold answers. MCTS-DPO estimates step values with tree search and runs online step-level DPO on GSM8K, MATH and ARC. Both papers report factual hallucination in long chains, and neither retrieves anything. Iterative RPO builds trajectory-level pairs from gold-answer correctness. It found that plain DPO can lower the likelihood of both the winning and losing answers, and fixed this by adding a likelihood term on the winner. It also reports that gains plateau after 3–4 iterations, which it attributes to the lack of new knowledge. We reuse its likelihood anchor, and we test whether retrieval moves that plateau.
On the retrieval side, several systems already optimize multi-hop agents, but at a different granularity or with heavier supervision. EVO-RAG trains only a query rewriter with DPO. It scores rewrites with a seven-factor reward model and uses gold documents and answers. GRAT runs tree search over sub-questions and retrievals, then fine-tunes only on successful paths, discarding the failed branches. Q-PRM trains a separate process reward model for pre-retrieval rewriting and consults it at inference, which adds latency. Q-RAG freezes the generator and trains the retrieval embedder with Q-learning. It requires sentence-level support-fact labels and names LLM-derived rewards as future work.
Our proposal differs from all four in two ways. It trains the reasoning-and-search policy itself on contrastive hop pairs. It also treats the gold-label requirement as a quantity to minimize rather than an assumption.
Two papers constrain the design. DMPO shows that DPO's derivation breaks for multi-turn trajectories, because a normalizing term (the partition function) depends on the state. We sidestep this rather than solve it. Every pair we build shares the entire prefix, including the retrieved documents, and differs only in the next hop. The normalizer depends only on that shared state, so it cancels exactly, as in single-turn DPO. The second constraint comes from SSPO and RLTHF. Both show, for dialogue and summarization, that a small labeled seed plus thresholding or targeted querying recovers most of the value of full annotation. They also warn that self-judging loops amplify their own biases. We borrow their idea of calibrating on a seed and spending labels where confidence is lowest, and we measure pseudo-label precision directly so that such a loop would be caught.
Research Program
The agent. The policy follows an IRCoT-style loop, after IRCoT, which alternates a reasoning sentence with a retrieval keyed on that sentence. We make one change: the model emits an explicit subquery with each reasoning sentence, and it emits an explicit answer action when it is done.
Building hop pairs. For each training question, we sample a trajectory with up to four hops. At each hop $t$ with shared state $s_t$ (the question, earlier hops and retrieved passages), we sample $M=3$ alternative actions. From each action we resume $K=4$ completions, which gives a value estimate $\hat v(s_t,a)$, the fraction of resumptions judged correct.
We then locate the first hop where some sibling action beats the taken action by a margin $\delta$. The taken action becomes the loser $a_l$ and the best sibling becomes the winner $a_w$. This is Step-DPO's first-error localization, with Monte Carlo resumption in place of a math checker. Retrieved passage tokens are environment tokens and are excluded from the loss.
The objective. The loss is DPO restricted to the hop, plus the anchor from Iterative RPO:
$$\mathcal{L}=-\log\sigma\!\Big(\beta\log\tfrac{\pi_\theta(a_w\mid s_t)}{\pi_{\text{ref}}(a_w\mid s_t)}-\beta\log\tfrac{\pi_\theta(a_l\mid s_t)}{\pi_{\text{ref}}(a_l\mid s_t)}\Big)-\lambda\,\tfrac{1}{|a_w|}\log\pi_\theta(a_w\mid s_t).$$
Here $\pi_\theta$ is the policy being trained and $\pi_{\text{ref}}$ is the frozen starting policy. $\beta$ sets how far the policy may move from $\pi_{\text{ref}}$, and $\lambda$ weights the likelihood anchor on the winner, which is normalized by winner length $|a_w|$. Training runs for three to five iterations, and pairs are rebuilt on-policy each round.
The scarce-label regime. "Judged correct" means one of three things, and each defines an experimental condition.
In the full-gold condition, a completion is correct if it matches the benchmark answer.
In the seed-plus-agreement condition, gold answers are revealed for only $b\%$ of questions. For every question we compute the agreement rate of the modal answer across all resumed completions. On the labeled seed, we fit a threshold on agreement above which the modal answer is correct with at least 90% precision. This is a simple analogue of SSPO's calibrated threshold. On unlabeled questions above the threshold, resumptions that match the modal answer count as correct. Questions below the threshold contribute no pairs.
The active condition uses the same budget, but the $b\%$ labels go to the questions with the lowest agreement instead of a random sample, following RLTHF's logic of labeling where the automatic signal is least trustworthy.
We simulate annotation by withholding benchmark gold answers. This is honest about cost only in the sense of counting labeled questions; we report cost as a label count.
Baselines
The naive reference point is the base instruction-tuned model, Llama3-8B, run in the same interleaved loop with IRCoT-style few-shot prompting and no training. IRCoT reports that this style of prompting improves QA F1 by up to 15 points over single-step retrieval. It is the zero-training floor that every trained method must clear.
The strongest existing competitor on the same model and benchmark is GRAT-style self-training: tree search over the agent's own trajectories, followed by fine-tuning on high-reward paths. GRAT reports that Llama3-8B on 2WikiMultiHopQA rises from 60.6% to 66.8% accuracy after fine-tuning on 11,000 GRAT trajectories. We will reproduce this pipeline with our retriever and corpus, with the number of trajectory rollouts matched to our total resumption budget. Beating it means exceeding its reproduced accuracy on 2WikiMultiHopQA and MuSiQue by a margin larger than the seed-to-seed spread over three training seeds, at equal generated tokens.
The second strong competitor is trajectory-level preference learning. This is Iterative RPO's DPO-plus-likelihood objective applied to whole interleaved trajectories, with winners and losers chosen by final-answer correctness, the same iterations and the same rollout budget. It isolates our key ingredient. If hop-level pairs do not beat whole-trajectory pairs at matched compute, the localization idea has failed.
EVO-RAG is the closest DPO-for-multi-hop system. It reports Exact Match gains of up to 4.6 points over strong RAG baselines with 15% shallower retrieval, but the file gives no absolute figure. Since EVO-RAG tunes only a rewriter and needs a trained reward model, we compare against its reported deltas relative to our prompted baseline rather than reimplement it.
Three ablations complete the design. The first removes the likelihood anchor ($\lambda=0$) to test whether the collapse observed by Iterative RPO appears in retrieval traces. The second replaces branch-and-resume values with a single outcome label propagated to all hops. The third applies DMPO-style length normalization to whole trajectories, to test whether shared-prefix pairing is really what avoids the multi-turn problem.
For every method, tuning gets the same budget: the same grid over $\beta$ and learning rate, and the same number of preference pairs or fine-tuning examples.
Why not just…?
Why not just fine-tune on successful trajectories, as GRAT does? It is simpler, and it is the main threat. PORT found that fine-tuning on correct traces overfits in math, and contrastive DPO on flawed traces gave up to an 18.73% relative gain on AQuA-RAT. That evidence comes from closed-world arithmetic, though, and may not transfer. Our comparison uses the same resumption budget, so the failed branches that GRAT discards are the only extra information we exploit. If fine-tuning matches us, the negative signal is not worth its cost here, and we will report that.
Why not just apply Iterative RPO to whole trajectories? The benchmarks provide gold answers, and trajectory pairs need no branching. This alternative could work nearly as well on two-hop questions, where localization matters little. We therefore report results split by hop count, using MuSiQue's 2–4 hop structure. Our prediction is that hop-level pairs pull ahead on 3–4 hop questions. If they do not, the extra rollouts are wasted.
Why not just use self-consistency voting at inference, with no training? Majority voting over sampled trajectories needs no labels. Iterative RPO's own GSM8K numbers show voting adding substantially on top of training: 81.6% to 88.7%. We will report voting with the same sample count, applied both to the base model and to our trained model. Training has to beat voting on the base model, or at least compose with it, to justify itself. Because voting costs inference compute, we report accuracy against inference tokens.
Why not just freeze the generator and train the retriever, as Q-RAG does? Q-RAG needs support-fact labels, which our scarce-label regime explicitly lacks. MultiHop-RAG shows that even with ground-truth evidence supplied, Llama2-70B reaches only 0.32 accuracy and Mixtral-8x7B 0.36, so better retrieval alone cannot fix reasoning over the retrieved evidence. On that benchmark we will run our trained policy with gold evidence supplied, to check whether the reasoning half improves on its own.
Proposed Experiments and Decisive Checks
We evaluate on HotpotQA (paper) in the full-Wikipedia setting, MuSiQue-Ans (paper), 2WikiMultiHopQA and Bamboogle, all with a single fixed retriever over Wikipedia. We report EM and F1. Two context points: MuSiQue humans reach 78.0 answer F1, and the file has no reported result on Lacuna for GRAT or EVO-RAG absolute scores on HotpotQA or Bamboogle. MultiHop-RAG's null-query category tests whether training raises hallucination on unanswerable questions.
The first decisive check is hop-level versus trajectory-level DPO at matched compute, split by hop count.
The second is the label-efficiency curve. We run $b\in\{1,5,10,25,100\}\%$ for both random-seed and active labeling. We report final accuracy and, separately, pseudo-label precision measured against the withheld gold answers. The label claim holds if active labeling at 5–10% recovers at least 90% of the full-gold gain over the prompted baseline, and pseudo-label precision stays above the calibrated target across iterations. Precision drifting downward across rounds would be direct evidence of the self-reinforcing loop that SSPO warns about, and it would falsify the second claim even if accuracy looked fine.
The third check is whether accuracy keeps rising past four iterations. If it does, that supports the claim that retrieval breaks the plateau reported by Iterative RPO. If it stalls like the closed-world case, the plateau is not about missing knowledge.
We also track average hops per question and retrieval calls, to see whether hop-level training reduces redundant searching the way EVO-RAG reports.
References
- Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs
- Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning
- Iterative Reasoning Preference Optimization
- Curriculum Guided Reinforcement Learning for Efficient Multi Hop Retrieval Augmented Generation
- GRAT: Guiding Retrieval-Augmented Reasoning through Process Rewards Tree Search
- Q-PRM: Adaptive Query Rewriting for Retrieval-Augmented Generation via Step-level Process Supervision
- Q-RAG: Long Context Multi‑Step Retrieval via Value‑Based Embedder Training
- Direct Multi-Turn Preference Optimization for Language Agents
- Semi-Supervised Preference Optimization with Limited Feedback
- RLTHF: Targeted Human Feedback for LLM Alignment
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
- PORT: Preference Optimization on Reasoning Traces
- MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries
- HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
- ♫ MuSiQue: Multihop Questions via Single-hop Question Composition