Search agents answer multi-hop questions such as "In what city was the director of Inception born?" by alternating between a reasoning step, a search query, and reading the passages that come back. Training them with reinforcement learning usually gives a single reward at the end, for whether the final answer was right. That tells the agent almost nothing about which of its five searches helped. Every retrieved passage also stays in the prompt, so irrelevant text piles up across hops and pulls the model toward confident wrong answers. We propose one self-supervised signal that addresses both problems. During training, for each hop, we measure how much the newly retrieved evidence raises the model's own probability of the known correct answer, compared with the same hop given substitute passages from the same search. This counterfactual gain becomes a per-hop reward. The same measurement, computed passage by passage, tells us which passages the model should have kept in its context and which it should have dropped.
What is new here is the counterfactual. Existing "confidence ascent" rewards credit a step for any rise in the model's belief in the answer, whether or not that rise came from the evidence. In retrieval, belief also moves for reasons unrelated to evidence: the model guesses from memory, or it reacts to distracting text. The question we test is whether conditioning the reward on the evidence gives search agents the benefit of tree-based process supervision at a fraction of its rollout cost, and whether it also makes them robust to injected distractors.
Core Claim
Let $a^*$ be the gold answer and $h_{\le k}$ the agent's history through hop $k$, where hop $k$ contains its query, the retrieved evidence $E_k$, and its sub-answer. Define the model's belief after hop $k$ as $b_k = \log \pi(a^* \mid x, h_{\le k})$, where $\pi$ is the current policy and $x$ the question. Let $\tilde b_k$ be the same quantity with $E_k$ replaced by counterfactual passages $\tilde E_k$, which are lower-ranked results from the same query, matched for length. The per-hop reward is
$$r_k = \underbrace{(b_k - b_{k-1})}_{\text{belief progress}} \;-\; \beta \,\underbrace{(\tilde b_k - b_{k-1})}_{\text{progress without the evidence}},$$
where $\beta \in [0,1]$ interpolates between plain confidence ascent ($\beta=0$) and pure evidence attribution ($\beta=1$, giving $r_k = b_k - \tilde b_k$).
For each passage $j$ in $E_k$, the leave-one-out gain $g_{k,j} = b_k - b_k^{(-j)}$ labels that passage as useful or as noise. We then train the agent to emit an explicit keep or drop decision for each passage, so that at test time, when $a^*$ is unknown, the agent prunes its own context.
The claim is that $r_k$, added to the outcome reward inside GRPO (Group Relative Policy Optimization, a reinforcement learning method that scores each sampled answer against the other samples for the same prompt), will do three things. It should match the multi-hop accuracy of Monte Carlo tree process supervision. It should need only prefill passes, which read text without generating any, rather than extra rollouts. And under distractor injection it should degrade less than both outcome-only training and uncounterfactualized confidence ascent.
Why This Gap Exists
Three strands of prior work motivate this design.
The first shows that the model's belief in the gold answer is a usable dense reward. PACR rewards stepwise increases in $\log P(Y_{gt}\mid \text{step}_{\le k})$ and lifts Qwen2.5-Math-7B from 30.0% to 43.3% on AIME 2024. RePro adds a smoothness term for the same probability gain. ΔBelief-RL carries belief-change rewards to long-horizon interaction. We reuse this technique, but all of these works score the reasoning text itself. None asks whether a belief change came from external evidence, which is the central question once each step injects retrieved documents.
The second strand explains why that question matters. Cho et al. show that readers get worse as more documents are retrieved, and that they are highly confident when misled (0.98 on a wrong "Lord Voldemort"). Adaptive Distraction and Breaking Focus find that model-specific distractors cut accuracy by over 45%, that telling the model to "ignore irrelevant information" does almost nothing, and that the failure disappears once the distractor is removed. That last finding means the problem is how the model allocates attention, not what it knows. A reward that is blind to whether the evidence caused the belief change will therefore credit hops for noise-driven belief swings.
The third strand constrains the design. Chen et al. show that rewards built from the model's own confidence (entropy minimization) eventually collapse toward confidently wrong behavior. Our reward avoids that failure because it is anchored to the gold answer and kept alongside the outcome reward. We accept the limitation noted for PACR and implicit PRMs: gold answers are needed during training. Multi-hop QA training sets supply them.
The closest existing systems each lack one of our ingredients. TreePS-RAG estimates node values by averaging descendant leaf outcomes in an online search tree. It prunes sibling nodes by how much their retrieved document sets overlap lexically, using Jaccard similarity, which cannot tell a distracting document from a useful one, and it requires full rollouts. DPRM derives implicit process rewards for multi-hop QA but depends on the Freebase knowledge graph and ignores document-level noise. GRAT uses an external scorer inside tree search and then trains offline by fine-tuning on successful paths. Q-PRM supervises only query rewriting that happens before any retrieval. HANRAG targets noise across hops with a routing agent, but its entry in our file is incomplete, so we treat it as a baseline rather than characterize it further. None of these computes a per-hop, evidence-counterfactual belief gain, and none reuses one signal for both credit assignment and learned context pruning.
Research Program
The agent follows the interleaved reason-search-read loop of IRCoT and CoRAG. We add one output format: after reading each passage, the agent emits a short keep or drop tag. Dropped passages are removed from the context for all later hops.
Training is GRPO on Qwen2.5-7B-Instruct, the backbone for which TreePS-RAG reports results. The objective combines three signals: the outcome reward (exact match of the final answer), the hop rewards $r_k$ spread over the tokens of each hop, and a tagging reward that pays for agreement between each keep or drop tag and the sign of $g_{k,j} - \tau$, where $\tau$ is a fixed threshold.
Counterfactual passages come from ranks just below the top-$m$ results of the same query. This keeps topic and style constant and changes only whether the passage is informative. As a harder variant, we generate self-authored distractors: Adaptive Distraction shows that models are good at producing distractors aimed at themselves.
Every reward term needs only forward passes that score the fixed string $a^*$ given a prefix. Per hop, that is one pass for $b_k$, one for $\tilde b_k$, and $m$ leave-one-out passes, with the key-value cache (stored attention states from earlier tokens) shared across the common prefix. We will report wall-clock time per training step against TreePS-RAG's tree expansion, since the training cost of tree search, roughly 2x for TreeRL, is the main practical argument for our signal.
The analysis side asks when belief progress and evidence attribution disagree. We will bucket hops by the sign pattern of $(b_k - b_{k-1})$ and $(b_k - \tilde b_k)$, and inspect hops with rising belief but no evidence attribution. We predict these are mostly parametric guesses and distractor effects. If instead they turn out to be mostly legitimate bridge hops, whose evidence only pays off at a later hop, the per-hop counterfactual is too myopic. In that case we would extend it to credit $E_k$ with the gain in all later beliefs $b_{k'}$ for $k' > k$.
Baselines
The naive reference is the same agent loop with no training. It is prompted IRCoT-style on the same retriever and corpus, with every retrieved passage kept, and it establishes how much any training helps.
The outcome-only reference is Search-R1-style GRPO with an exact-match reward and the same data, backbone, retriever, and number of policy updates. We found no reported Search-R1 number on Lacuna beyond TreePS-RAG's statement that it outperforms Search-R1, so we will run this baseline ourselves.
The strongest existing method is TreePS-RAG. It reports a 49.0% average score across NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultihopQA, Bamboogle, and MuSiQue on Qwen2.5-7B-Instruct. It is the strongest competitor because it already provides annotation-free per-hop credit and some control over redundant retrieval. We will reproduce it on our retriever and match on two budgets: the same number of policy updates, and the same total GPU-hours. Our method is cheaper per step, so under matched GPU-hours it gets more updates. We count it as beating TreePS-RAG if it matches TreePS-RAG's clean average within one point at no more than half the training GPU-hours, and clearly exceeds it under distractor injection.
For noise robustness specifically, we include HANRAG. No reported result for it was found on Lacuna, because its entry in our file is truncated, so we will run it ourselves on identical inputs. We also include a DPO model trained on distracted-versus-clean reasoning pairs, following Adaptive Distraction.
Two ablations isolate our key ingredient. Setting $\beta=0$ gives plain PACR-style confidence ascent applied to retrieval hops. Removing the tag head while keeping $r_k$ separates the contribution of credit assignment from that of context pruning.
Why not just…?
Why not just use outcome reward with more rollouts? Sparse terminal reward cannot say which hop went wrong. The TreePS-RAG ablation shows that removing its process advantage drops performance to Search-R1 levels. The outcome-only baseline, given our method's matched GPU-hours, directly tests whether extra samples can substitute for dense credit.
Why not just use plain confidence ascent, as PACR does? This is the real threat. If $\beta=0$ matches $\beta=1$ on both clean and distracted data, the counterfactual adds nothing, and the contribution reduces to "PACR transfers to RAG." The distractor-injection experiment settles it. We predict that $\beta=0$ credits hops whose belief rises for distractor-driven or parametric reasons, and so degrades more under injection. The disagreement analysis will show whether such hops actually occur.
Why not just add a reranker or an extract-then-generate filter, like Ext2Gen? Ext2Gen and rerankers judge relevance to the query. They do not judge whether a passage moves this particular model toward the answer, and they give no credit signal for the search policy. We will stack a reranker onto the outcome-only baseline. If that combination closes the robustness gap, our pruning head is redundant and only the credit-assignment claim survives. We would report that outcome as such.
Why not just fine-tune against distractors with DPO? Adaptive Distraction reports that this helps small models by nearly 50% on perturbed inputs. However, it trains on a fixed set of perturbations and gives no per-hop credit. It is included as a baseline and evaluated on distractor types it was not trained on.
Proposed Experiments and Decisive Checks
We evaluate on the seven-benchmark suite used by TreePS-RAG, with emphasis on MuSiQue, which is built to block shortcuts, 2WikiMultihopQA, and HotpotQA. Metrics are exact match and F1, retrieval recall of the gold supporting passages, and context tokens carried per question. For reference points, CoRAG reports MuSiQue recall rising from 29.0% with single-step retrieval to 47.1%. DRAGIN reports an F1 of 0.3931 on 2WikiMultihopQA with LLaMA2-13B. GRAT reports that fine-tuning Llama3-8B raises 2WikiMultihopQA accuracy from 60.6% to 66.8%.
Robustness is tested at three levels of difficulty. First, we add the dataset's own distractor paragraphs to every hop. Second, we add same-type-entity passages mined from the corpus. Third, we add self-generated adversarial distractors. For each level we report accuracy drop relative to the clean setting, and the confident-wrong rate: the fraction of wrong answers the model assigns probability above 0.9, which measures the failure Cho et al. document.
Three results would decide the proposal. It is falsified if $\beta=1$ fails to beat $\beta=0$ under injection. It is falsified if matched-GPU-hour TreePS-RAG beats it on clean multi-hop average. And it is falsified if the learned keep or drop tags agree with held-out leave-one-out labels only at the level of a query-relevance reranker, and the pruned context does not reduce the confident-wrong rate. Success means comparable clean accuracy to tree supervision at lower cost, a smaller degradation curve under all three distractor levels, and shorter contexts. We would regard that as evidence that the question "did the evidence change the model's mind?" is the right unit of credit for retrieval agents.
References
- PACR: Progressively Ascending Confidence Reward for LLM Reasoning
- Rectifying LLM Thought from Lens of Optimization
- Intrinsic Credit Assignment for Long Horizon Interaction
- Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering
- Adaptive Distraction: Probing LLM Contextual Robustness with Automated Tree Search
- Breaking Focus: Contextual Distraction Curse in Large Language Models
- Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection
- Free Process Rewards without Process Labels
- TreePS-RAG: Tree-based Process Supervision for Reinforcement Learning in Agentic RAG
- DPRM: A Dual Implicit Process Reward Model in Multi-Hop Question Answering
- GRAT: Guiding Retrieval-Augmented Reasoning through Process Rewards Tree Search
- Q-PRM: Adaptive Query Rewriting for Retrieval-Augmented Generation via Step-level Process Supervision
- HANRAG: Heuristic Accurate Noise-resistant Retrieval-Augmented Generation for Multi-hop Question Answering
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
- Chain-of-Retrieval Augmented Generation
- TreeRL: LLM Reinforcement Learning with On-Policy Tree Search
- Aligning Extraction and Generation for Robust Retrieval-Augmented Generation
- ♫ MuSiQue: Multihop Questions via Single-hop Question Composition
- HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering