Multi-hop question answering asks a model to chain facts that sit in different documents. "In what city was the director of Inception born?" requires first finding the director, then searching for where that person was born. Language models trained with reinforcement learning on this task tend to learn shortcuts. They reach the right answer through the wrong evidence, and the only known fix is expensive: score every intermediate step during training with a large language model acting as judge. We propose instead to learn the step-level reward from expert demonstrations using inverse reinforcement learning (IRL), meaning we infer the objective an expert is optimizing from examples of the expert's behavior. Multi-hop datasets already contain such demonstrations, because they ship gold decompositions and supporting paragraphs for every hop.
The usual obstacle to IRL is cost. Every update to the candidate reward requires re-solving a full reinforcement learning problem. Swamy et al. showed that this cost comes mostly from global exploration. It can be avoided by restarting the learner from intermediate states taken from the demonstrations, a technique called expert resets. The technique has only been tested in robot simulators, where it carries two known weaknesses. Resets are physically hard to perform, and errors compound quadratically with the length of the task. Multi-hop retrieval removes both weaknesses. A text state, consisting of the question, the passages retrieved so far, and the reasoning so far, can be restored exactly by copying a prompt. Horizons are also short, two to four hops on MuSiQue, so quadratic compounding has little room to grow. We argue that multi-hop reasoning is the setting where reset-based IRL should work best, and we will test that claim directly.
Core Claim
A small reward model trained by comparing expert actions with learner actions at the same expert-reset state can replace an online judge as the source of step-level faithfulness signal. The policy trained against it should match the faithfulness of judge-based training while using a fraction of the training compute, and should beat heuristic and outcome-only rewards on how often a correct answer rests on the wrong evidence.
What is new is the combination of two things. The first is learning a per-hop reward from demonstrations rather than hand-writing it. The second is the reset trick, which makes that learning local and cheap. This trick also turns IRL's adversarial comparison into a paired comparison: expert and learner act from the identical state, so the comparison has much lower variance than one between whole trajectories.
Positioning Against Prior Work
The work that motivates this proposal is the forward-RL line on multi-hop QA. CRAFT documents the "right-answer-wrong-reason" failure. It fixes the failure with Group Relative Policy Optimization (GRPO, a policy-gradient method that scores each sample against a group of samples for the same prompt), using a hand-weighted mix of rule checks and an online LLM faithfulness judge. That judge cost kept CRAFT to at most 20k training samples and 7B models. FrugalRAG trains adaptive stopping with a hand-crafted penalty around an oracle hop count. It is efficient, but it leaves intermediate reasoning unrewarded. Earlier RL search agents, such as Learning Open Domain Multi-hop Search Using Reinforcement Learning and reward shaping for knowledge-graph reasoning, likewise rely on engineered or terminal rewards. All of these choose the reward by hand. We infer it.
The work that supplies our method comes from two sources. Swamy et al. contribute expert resets and the FILTER interpolation between reset starts and full rollouts. Fu et al. contribute evidence that IRL on text can recover dataset-specific trade-offs better than hand-tuned weights; on CNN/DM, ROUGE-L rose from 41.24 to 43.15. Fu et al. score whole summaries in a single turn, with no notion of hops or retrieval actions. Swamy et al. never left continuous control.
One paper constrains our design. Q-RAG trains a multi-step retriever cheaply with soft Q-learning, using support-fact labels as the reward. It shows that dense per-step supervision makes retrieval learnable on one GPU. We keep that dense structure but replace the labels with a learned reward. Unlike labels, a learned reward can be reused where no labels exist.
Research Program
We first build expert trajectories. On MuSiQue we use the single-hop decompositions and supporting paragraphs (Trivedi et al.). On 2WikiMultiHopQA we use evidence tuples (Constructing A Multi-hop QA Dataset). Each hop $k$ becomes an expert state $s_k^*$ and an expert action $a_k^*$.
- Expert state $s_k^*$: the question, the gold passages and intermediate answers for hops before $k$, and the retrieval results for those hops.
- Expert action $a_k^*$: the sub-query, the cited passage, the intermediate answer, or the decision to stop.
The policy is a 7B open model in a structured trace format similar to CRAFT's. It uses the same retriever family as FrugalRAG, either ColBERT or E5.
Reward learning happens at reset states. We sample $s_k^*$, draw a learner action $a \sim \pi(\cdot \mid s_k^*)$, and fit a small encoder reward $r_\phi$ with a paired loss:
$$\mathcal{L}(\phi) = -\,\mathbb{E}_{(s_k^*, a_k^*) \sim \mathcal{D}_E,\; a \sim \pi(\cdot \mid s_k^*)} \log \sigma\big(r_\phi(s_k^*, a_k^*) - r_\phi(s_k^*, a)\big).$$
Here $\mathcal{D}_E$ is the demonstration set and $\sigma$ is the logistic function. Some learner actions are equivalent to the expert's, for example when they retrieve the same gold paragraph. We drop those pairs so the reward does not penalize correct paraphrases.
Policy learning also starts from reset states. We run GRPO from a mixture of starts: with probability $\alpha$ the episode begins at a reset state $s_k^*$ with $k$ drawn uniformly, and otherwise it begins at the question. The objective is
$$J(\pi) = \mathbb{E}\Big[\textstyle\sum_{t \ge k} r_\phi(s_t, a_t) + \lambda\, \mathrm{F1}(\hat y, y)\Big],$$
where $\hat y$ is the predicted answer, $y$ the gold answer, and $\lambda$ weights answer correctness. Reward updates and policy updates alternate.
We then analyze three things. The first is how the best $\alpha$ and the policy's error depend on hop count. The second is what the learned reward attends to; we probe it against support-fact overlap, sub-query specificity, and stopping. The third is whether a frozen $r_\phi$ transfers to new datasets.
Baselines
The naive reference point is supervised fine-tuning on the same expert traces, which is behavioral cloning. It uses identical data and no rollouts. CRAFT reports that this approach often degrades faithfulness, and our method must clearly exceed it on faithfulness and on two-to-four-hop accuracy.
The strongest efficiency competitor is FrugalRAG. We re-run it with its published recipe of about 1,000 HotpotQA examples, on the same base model and retriever. Its reported HotpotQA result is 68.47% on model-based evaluation with 2.05 searches, against 64.20% with 2.76 searches for ReAct (FrugalRAG). Beating it means matching or exceeding its accuracy at a comparable search count while lowering the right-answer-wrong-reason rate.
The strongest faithfulness competitor is CRAFT, with its online judge and the same structured template, trained on the same 20k mix. CRAFT reports +17.50% EM and up to +22.1% faithfulness on MuSiQue for its 7B model over the base model (CRAFT). Our target is to reach CRAFT's faithfulness within a few points while using an order of magnitude less training-time judge compute. Matching it at equal compute would not count as a win.
Two ablations isolate our ingredients, and they share our data, rollout budget, and tuning budget:
- Reset-only GRPO: the same reset curriculum, but only the answer-F1 reward. This removes the learned reward.
- Full-rollout adversarial IRL: the same reward model trained on whole learner trajectories from the question ($\alpha = 0$). This removes the resets.
Q-RAG serves as a retrieval-only reference on MuSiQue and HotpotQA, run from its released code. We found no reported Q-RAG result on Lacuna for HotpotQA or MuSiQue, so we will report our own run.
Why not just…?
Why not just do reset-curriculum GRPO with answer reward and skip IRL? This is the most serious threat to the proposal. Starting from intermediate hops may by itself fix most of the exploration problem and recover answer accuracy. If it also recovers faithfulness, our learned reward is unnecessary. The reset-only ablation tests exactly this, and we commit in advance to the result. If reset-only GRPO matches us on support-fact F1 and on the wrong-reason rate, the contribution shrinks to "resets help multi-hop RL." CRAFT's finding that outcome-only optimization produces shortcuts is our reason to expect a gap, but CRAFT did not test reset starts.
Why not just distill CRAFT's judge into a small reward model offline? This is also a real competitor, because it too removes online judge calls. The weakness is that the judge labels are collected on one policy's outputs and go stale as the policy drifts, unless the judge is called again. Our paired loss needs only demonstrations and fresh learner samples. We will run judge distillation with a judge-call budget matched to the human-free cost of our method. If periodic re-labeling closes the gap cheaply, we will say so.
Why not just supervised fine-tuning on the gold traces? It is the cheapest option, and it is our naive baseline. CRAFT reports that it degrades faithfulness. FrugalRAG's supervised-only stage searched 5.99 times per question on HotpotQA, which shows that imitation alone does not learn when to stop. We will additionally measure accuracy as a function of hop count, to see whether its errors compound on four-hop MuSiQue questions.
Why not just prompt a large model with interleaved retrieval? IRCoT works without any training. However, on BrowseCompPlus a 600B DeepSeek-R1 reached only 16.39%, against over 20% for a trained 7B model (FrugalRAG). We include IRCoT-style prompting of our base model as a reference.
Proposed Experiments
We train on a MuSiQue plus 2WikiMultiHopQA mix and evaluate on all three standard datasets, including held-out HotpotQA.
- Metrics: EM and F1; supporting-paragraph F1 against gold annotations; the right-answer-wrong-reason rate, meaning answers that are correct but cite non-gold evidence; and CRAFT's judge-based faithfulness score.
- Judge validity: the judge is used for evaluation only, and we re-validate it on a human-labeled subsample. CRAFT reports Cohen's κ = 0.78 on MuSiQue.
- Cost: training compute is reported as rollout tokens, judge calls, and GPU-hours.
Three checks decide whether the proposal succeeds.
- Faithfulness at matched compute. The learned reward must beat reset-only GRPO on the right-answer-wrong-reason rate. Otherwise the IRL component is not justified.
- The horizon claim. We sweep $\alpha$ and break results down by MuSiQue hop count. The claim predicts that pure resets ($\alpha = 1$) remain competitive at two and three hops and degrade only at four. If they degrade sharply even at short horizons, then error from distribution shift in text, not horizon length, dominates, and the argument for resets fails.
- Reward transfer. We freeze $r_\phi$ and train a fresh policy on HotpotQA without its support facts. We then evaluate zero-shot on BrowseCompPlus, where FrugalRAG reports over 20%. A reward that transfers is the main practical advantage IRL offers over support-fact supervision of the kind Q-RAG uses.
References
- Inverse Reinforcement Learning without Reinforcement Learning
- CRAFT: Calibrated Reasoning with Answer-Faithful Traces via Reinforcement Learning for Multi-Hop Question Answering
- FrugalRAG: Less is More in RL Finetuning for Multi-hop Question Answering
- Learning Open Domain Multi-hop Search Using Reinforcement Learning
- Enhancing Multi-Hop Knowledge Graph Reasoning through Reward Shaping Techniques
- Inverse Reinforcement Learning for Text Summarization
- Q-RAG: Long Context Multi‑Step Retrieval via Value‑Based Embedder Training
- ♫ MuSiQue: Multihop Questions via Single-hop Question Composition
- Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
- HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering