Multi-hop questions need several pieces of evidence chained together, such as "Who directed the film that won Best Picture the year X was born?" Systems that answer them by retrieving documents and reasoning step by step tend to fail in a particular way. One reasoning step quietly stops relying on the retrieved evidence and fills the gap from memory or from earlier text. Every later step and every later search query is then built on that unsupported claim. Current systems either cannot see this happen or watch for it with weak signals: how uncertain the next word is, or how much raw attention (the weights a transformer uses to mix information across positions) lands on each passage. We propose a per-step grounding check. After each reasoning sentence, run one backward pass. Score each retrieved passage by gradient-weighted attention, which means the attention weights multiplied element-wise by the gradient of the step's own log-probability with respect to those weights. This product is signed, so it says whether a passage pushed the model toward what it just wrote or against it.
The check then drives a control loop. If a step draws little positive support from retrieved evidence, or draws strong negative support (it contradicts what it attended to), the step is rejected, a corrective retrieval is issued, and the step is regenerated. The opportunity rests on two findings. Raw attention is a poor and often sign-blind measure of what drives a prediction. Gradient-weighted saliency, by contrast, has been shown to collapse just before hallucinations in vision-language models. No one has tested whether that signal can find and repair broken hops in text retrieval chains.
Core Claim
The claim is that a step-level, signed, gradient-weighted grounding score is better than three cheaper signals at predicting which intermediate reasoning steps are wrong in retrieval-augmented multi-hop question answering. Those signals are token entropy, raw attention mass on evidence, and a single with/without-context forward comparison. A further claim is that gating re-retrieval on this score improves final answer accuracy at a compute cost close to iterative baselines. The cost stays close because the backward pass runs once per reasoning sentence (three to five per question), not once per token. The new element is the unit and the use. The unit is one reasoning step scored against each retrieved passage, with sign kept. The use is a decision made inside the generation loop: accept the step, or reject it and retrieve again. It is not post-hoc citation.
Concretely, let step $s$ have generated tokens $y_s$, and let $\mathcal{L}_s = -\log p(y_s \mid \text{context})$ be the model's loss on its own output, which needs no gold label. For layer $l$ and head $h$, let $\alpha^{(l,h)}_{ij}$ be the attention from position $i$ to position $j$. The signed contribution of passage $p$ to step $s$ is
$$A_{s,p} = \sum_{l,h}\ \sum_{i \in s}\ \sum_{j \in p} \alpha^{(l,h)}_{ij} \cdot \left(-\frac{\partial \mathcal{L}_s}{\partial \alpha^{(l,h)}_{ij}}\right).$$
Here $i$ ranges over the step's token positions and $j$ over the passage's positions. Positive $A_{s,p}$ means the passage supports the step. Negative means it acts as a suppressor. The grounding ratio $G_s$ is the share of total positive contribution that comes from retrieved passages rather than from the question, earlier steps, or the instruction prefix. The conflict score $C_s$ is the largest negative $A_{s,p}$. A step is accepted when $G_s$ is high and $C_s$ is small. Both thresholds are tuned on a held-out split.
What This Builds On and Departs From
Three papers motivate the signal. Is Attention Interpretable? erases attention components and shows that zeroing the top-weighted one flips decisions in only about 9% of cases, while gradient rankings find flips at least 1.6 times faster. Rethinking Attention-Model Explainability through Faithfulness Violation Test shows that raw attention ignores sign. Up to 38% of top-attended regions in visual question answering are suppressors, and attention-times-gradient cuts the violation ratio from 30–40% to under 7%. That paper's result is the reason we keep the sign rather than taking magnitudes. Hallucination Begins Where Saliency Drops supplies the measure we reuse, $\alpha \odot \nabla_\alpha \mathcal{L}$. It finds that saliency toward prior context collapses before vision-language hallucinations and uses this to reject candidate tokens during sampling. We depart from it in three ways. We move from images to retrieved text chains. We measure saliency toward external passages rather than toward the model's earlier outputs. We act once per sentence rather than once per token, which targets the 30–40% latency overhead and the memory limits that paper reports.
Three lines of work occupy nearby ground without doing this. DRAGIN decides when to retrieve using token entropy weighted by forward attention influence, and builds queries from the top forward-attended tokens. It never takes gradients, so it cannot tell a supporting passage from a suppressing one, and it does not check whether a finished step was grounded. MIRAGE and ARC-JSD attribute answers to context using model internals. MIRAGE uses gradient saliency on context-sensitive tokens; ARC-JSD ablates sentences and measures Jensen–Shannon divergence. Both operate after generation and do not steer the reasoning chain. ARC-JSD does reach multi-hop data, and its finding that context use is concentrated in middle and upper heads guides which layers we aggregate over. Back Attention diagnoses "hopping too late" in closed-book reasoning and fixes it by changing the architecture. Our setting externalizes the hops into retrieval and leaves the model unchanged.
The host pipelines come from iterative retrieval. IRCoT alternates one reasoning sentence with one retrieval and gains 11–21 recall points and up to 15 F1. Chain-of-Retrieval Augmented Generation trains an 8B model on rejection-sampled retrieval chains and documents how a distractor retrieved at one hop sends the whole chain into hallucination. That failure is the one our check is meant to catch.
The paper that most constrains us is the vision-language saliency work. It reports that models trained with reinforcement learning from human feedback sometimes make high-saliency, confident errors. In our setting the analogue is a step that is firmly grounded, but in a distractor passage. Our score cannot flag such a step, because the step really is supported by evidence; the evidence is simply wrong. We therefore treat this failure type as outside the method's reach and measure how often it occurs.
Research Program
The work has three stages. The first is a diagnostic study that needs no control loop. We run an IRCoT-style pipeline with Llama-3.1-8B-Instruct, which matches CoRAG's base model, and with LLaMA-2-13B, which matches DRAGIN's reported setting. The benchmarks are MuSiQue, 2WikiMultihopQA, and HotpotQA. We align each generated reasoning sentence to a gold hop. MuSiQue provides decompositions and a gold paragraph for each sub-question, and 2WikiMultihopQA provides evidence triples. These annotations let us label each step as correct, ungrounded, or distractor-supported. We then ask two questions. First, does $A_{s,p}$ rank the gold paragraph for that hop first more often than raw attention, input-times-gradient, or the AttInGrad-style product from An Unsupervised Approach to Achieve Supervised-Level Explainability in Healthcare Records? Second, how well do $G_s$ and $C_s$ predict step errors, measured by area under the ROC curve?
The second stage is the control loop. A step that fails the check is discarded. The step's own text is issued as a new query, and passages with strongly negative $A_{s,p}$ are removed from the context. The step is then regenerated, with at most two retries per step. Across retries, the number of backward passes per question is capped and reported.
The third stage is a cost analysis. We measure wall-clock time, peak memory, and model-call counts against each baseline. We also test a cheaper variant that restricts gradients to the middle and upper layers ARC-JSD identifies, to see whether truncated backpropagation keeps most of the signal.
Baselines
The naive reference points are retrieving once and generating, and plain IRCoT with no step check. Plain IRCoT is the fair "do nothing" comparison because it shares our retriever, prompts, and step granularity.
The strongest training-free competitor is DRAGIN, with its entropy-plus-attention trigger and its attention-based query builder. On 2WikiMultihopQA with LLaMA-2-13B, DRAGIN reports F1 of 0.3931 against 0.3076 for FLARE. We will rerun DRAGIN with the same retriever, corpus, model, and threshold-tuning budget as our method, and also confirm that our rerun reproduces this number. Beating it means a higher F1 at matched or lower average model calls per question, and the gain must hold across three seeds of threshold selection.
CoRAG is the strongest trained system. The file reports it beating IterDRAG and Search-o1-32B by over 10 exact-match points on 2WikiQA and MuSiQue, and lifting MuSiQue retrieval recall from 29.0% to 47.1%. Because it is fine-tuned, we treat it as a ceiling rather than a head-to-head competitor. We will also apply our check to CoRAG's released-style chains, to test whether a trained planner still produces ungrounded steps that the check catches.
Three ablations isolate the key ingredients, and each keeps everything else fixed. The first replaces $\alpha \odot \nabla_\alpha \mathcal{L}$ with raw attention mass, which isolates the gradient. The second takes absolute values, which removes the sign and with it the conflict score. The third replaces the gradient with exact per-passage leave-one-out ablation, which tells us whether the first-order approximation loses anything.
Why not just…?
Why not just trigger on token entropy? Entropy says the model is unsure, but not where the uncertainty comes from. DRAGIN itself notes that entropy is high on function words, where retrieval would be wasted. Entropy is also a weak fit for the target failure: the step that derails a chain is often fluent and confident, because it was written from memory. The diagnostic study measures this directly. If entropy matches $G_s$ in area under the ROC curve for detecting ungrounded steps, the gradient is not needed and we will say so.
Why not just measure raw attention mass on the retrieved passages? It costs nothing beyond the forward pass, which makes it the most serious cheap rival. The evidence against it comes from classification and visual question answering, specifically the 30–40% violation ratios and the sign-blindness reported in Rethinking Attention-Model Explainability through Faithfulness Violation Test. That evidence may not carry over to long text contexts. The first ablation settles it. If raw mass detects step errors as well as the gradient score, the paper's conclusion becomes a negative transfer result for this line of interpretability work.
Why not just ablate context directly, as in MIRAGE's with/without comparison or ARC-JSD's sentence ablation? This is a real threat. A with/without-context forward pass costs only one extra call. With five passages, leave-one-out costs five calls, which may be cheaper than a backward pass in practice. The gradient's advantages are that it scores every passage, with sign, in one pass, and that it scales as context grows across hops. Whether those advantages matter is an empirical question. We will plot detection quality against measured cost for the gradient score, the with/without comparison, and per-passage ablation. If ablation dominates at the passage counts that occur in practice, we will adopt it as the recommended trigger and report the gradient only as an approximation that pays off in large-context regimes.
Why not just ask the model, or a natural-language-inference checker, whether the step follows from the passages? Self-verification needs no white-box access. On the other hand, MIRAGE found that prompted self-citation reaches only 30.6 citation F1 on ELI5, against 45.6 for internal attribution, which suggests self-reports describe grounding poorly. We still include a prompted self-check and an NLI verifier as step-error detectors. Each runs at matched compute and uses the same accept/reject loop.
Proposed Experiments and Decisive Checks
The main table reports answer exact match and F1 on MuSiQue-Ans, 2WikiMultihopQA, and HotpotQA under open retrieval. Two reference points from the file frame these numbers: MuSiQue human F1 is 78.0, and 2WikiMultihopQA's answer F1 upper bound is 91.8. Alongside answer quality we report retrieval recall of gold paragraphs, mean model calls, backward passes, latency per question, and peak memory. A second table gives step-error detection in area under the ROC curve for every signal discussed above, broken down by error type (ungrounded versus distractor-supported) using MuSiQue's per-hop gold paragraphs.
Three outcomes would be decisive. First, the gradient score must beat raw attention mass and entropy at detecting ungrounded steps by a clear margin, which we pre-register as at least 0.05 area under the ROC curve. If it does not, the central claim fails. Second, when the trigger is swapped into the same loop, end-to-end F1 must beat DRAGIN's reproduced 2WikiMultihopQA number at matched calls. Third, the sign ablation must hurt, because the conflict score earns its place only if removing passages with negative contribution helps beyond what the grounding ratio alone does. We will also report the share of errors that are distractor-supported, which this method cannot flag. That share sets the method's ceiling: if it dominates on MuSiQue, the remaining headroom belongs to better retrieval, not better grounding checks.
References
- Is Attention Interpretable?
- Rethinking Attention-Model Explainability through Faithfulness Violation Test
- Hallucination Begins Where Saliency Drops
- DRAGIN: Dynamic Retrieval Augmented Generation based on the Real-time Information Needs of Large Language Models
- Model Internals-based Answer Attribution for Trustworthy Retrieval-Augmented Generation
- Attributing Response to Context: A Jensen–Shannon Divergence Driven Mechanistic Study of Context Attribution in Retrieval-Augmented Generation
- Back Attention: Understanding and Enhancing Multi-Hop Reasoning in Large Language Models
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
- Chain-of-Retrieval Augmented Generation
- ♫ MuSiQue: Multihop Questions via Single-hop Question Composition
- Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps
- An Unsupervised Approach to Achieve Supervised-Level Explainability in Healthcare Records