Retrieval-augmented language models answer multi-step questions such as "In what city was the director of Inception born?" by retrieving a passage, working out an intermediate fact ("Christopher Nolan"), retrieving again, and so on. Almost every such system carries all previously retrieved text forward into every later step. That habit is costly. The prompt grows, and a distractor passage picked up at hop one can still derail the answer at hop three. The obvious fix is to throw away old evidence once the intermediate fact is known, but that is not always safe. If "the director" could be one of two people with the same name, or if the earlier passage carries a date or qualifier the next hop needs, discarding it breaks the chain.
We propose to treat this decision as a question of context-specific independence. That idea comes from probabilistic graphical models: some variable stops mattering for another, but only under particular values of a third, not under all of them. Concretely, the later reasoning step is independent of earlier evidence when the intermediate entity has been resolved to a single unambiguous value, and dependent otherwise. We will measure where this independence actually holds inside real language-model reasoning chains. We will train a small predictor that decides, per step, whether to delete the dependence on old evidence or keep it, and in the uncertain case branch over a handful of candidate values. The claim to test is that this conditional pruning beats both keeping everything and always pruning, on accuracy and on prompt length.
Core Claim
In a multi-hop chain, the evidence retrieved for earlier hops influences later hops mostly through the intermediate answers it produces. When an intermediate answer is resolved sharply, the edge from old evidence to later steps is empty and can be cut. When it is not resolved sharply, the edge carries real information and must be kept, or the ambiguity must be made explicit by branching.
Let $A_t$ be the model's answer at hop $t$, $E_{<t}$ the passages retrieved at earlier hops, and $B_{t-1}$ the intermediate answer from the previous hop. Our object of study is the context-specific statement
$$A_t \perp E_{<t} \mid B_{t-1}=b \quad \text{holds for some values } b \text{, not for all.}$$
We check it approximately, following the tolerance-based treatment of approximate independence in Zhang and Poole, who show that factorizing over only the configurations where a function is defined can speed exact inference by more than an order of magnitude when the independence is real. The operational test compares the model's answer distribution at hop $t$ with and without $E_{<t}$ in the prompt, holding the bridge value fixed. An edge counts as deletable when the divergence falls below a tolerance $\epsilon$.
What is new here is neither pruning context nor decomposing questions; both exist. The new parts are measuring when the pruning is licensed and making the pruning decision depend on the resolved values.
Relation to Prior Work
The mechanism comes from Boutilier et al.. They represent conditional probability tables as decision trees and show that, once a context is instantiated, "vacuous" edges can be deleted and ordinary graph-separation tests on the pruned graph reveal new independencies. They also introduce conditional cutset conditioning, which branches on a variable's values only where the evidence requires it. We carry both operations over to reasoning chains. Arc deletion becomes dropping earlier passages. Conditional cutsets become branching on the top candidates for an ambiguous bridge entity instead of on every step.
Shen, Choi, and Darwiche constrain the design. They show that axis-aligned decision-tree rules for contexts can need thousands of leaves where a learned function needs only a few contexts. For that reason our deletion rule is a small learned classifier over continuous signals, not a hand-written rule set.
On the retrieval side, several systems motivate the problem without solving it. IRCoT interleaves chain-of-thought steps with retrieval and appends every retrieved paragraph. It improves retrieval recall by 11–21 points and QA F1 by up to 15 points, but it is the clearest case of unbounded context growth. Li et al. document that long, distractor-heavy contexts degrade multi-hop reasoning. Their fix, fine-tuning a 7B model to cite or quote evidence at each step, operates on generation rather than on what the model is shown.
Three systems come closest.
ChainRAG rewrites each sub-question to replace unresolved references with the resolved entity. Its rewriting is unconditional: it always substitutes and never asks whether earlier context is still needed.
HANRAG routes questions into "compound" (independent sub-questions, run in parallel) versus "complex" (nested) types, and filters passages with a relevance discriminator. Its independence is a whole-question label assigned by a prompted classifier. Its filter asks whether a passage is relevant to the sub-question, not whether that passage is redundant given a resolved bridge. A passage can be relevant yet redundant, or can look irrelevant yet still be needed to disambiguate.
GRAT replaces linear chains with Monte Carlo tree search guided by process rewards. It branches everywhere and pays for rollouts. Our conditional branching is a much cheaper, targeted form of the same insurance.
Selective Induction Heads shows that transformers can learn, in context, to select which dependency structure is active, but only with at least three layers and only on synthetic Markov chains. We take this as weak evidence that the model's internal signals may reflect which structure is active. We do not rely on it.
Research Program
The work has three parts.
Measurement. On the training splits of MuSiQue-Ans (MuSiQue) and 2WikiMultiHopQA (2WikiMultiHopQA), we run an iterative retrieve-answer loop with Llama3-8B and Llama3-70B. At every hop after the first we produce paired counterfactual runs: one with the full history, one with only the rewritten sub-question and the resolved bridge value. We also run a third condition in which earlier passages are swapped for other passages that support the same bridge value. MuSiQue's gold decomposition graphs and 2Wiki's evidence triples tell us the true dependency skeleton, so we can separate cases where the skeleton says old evidence is irrelevant from cases where the model actually behaves that way.
The output is an empirical map of when the independence holds. We break it down by question shape, by hop, by how confident the bridge answer is, and by whether the bridge entity's name is ambiguous in the corpus. This map is itself a finding: nobody has reported how often later hops in LLM chains actually depend on earlier text beyond the extracted entity.
Deletion predictor. We train a small classifier on the measured labels. Its inputs are cheap: the bridge answer's token probability and entropy, the number of distinct candidate entities in the retrieved passages, the retrieval score gap between the top two candidates, and the hop type from the decomposition. When it predicts "deletable," the next step sees only the sub-question and bridge value. When it predicts "dependent," the system keeps the supporting sentences for the bridge, not all passages. When it predicts "dependent" and multiple candidates exist, the system branches over at most $m$ candidate bridge values and picks the answer by agreement and retrieval support, which is our conditional cutset. Because the inputs need only output probabilities, a variant using sampled-answer agreement instead of token probabilities will run on black-box APIs. This addresses the white-box restriction of attention-based methods like DRAGIN.
Stress test. We build a controlled ambiguity set by rewriting development questions so that the bridge entity has a same-named competitor, with disambiguating detail present only in the hop-one passage. On this set, always-prune should fail by construction and conditional pruning should not. It lets us verify that the predictor fires for the right reason.
Baselines
The naive reference points are one-shot retrieval and full-history iterative retrieval. One-shot retrieval retrieves top-$k$ once and answers. On HotpotQA's full-wiki setting its original baseline reaches only 17.73 joint F1 (HotpotQA), and ChainRAG reports roughly 60% higher average F1 than it on MuSiQue. Full-history iterative retrieval is IRCoT's loop. We rerun both ourselves with the same retriever, corpus, model, and decoding settings as our method, since the file reports IRCoT only as improvement ranges.
The most important ablation is always-prune. It is identical to our system except that the deletion predictor always says "deletable," which is essentially ChainRAG-style rewriting with history dropped. A second ablation, never-branch, keeps the predictor but disables candidate branching. It isolates the value of the cutset step from the value of deletion. A third, skeleton-oracle, deletes exactly where the gold decomposition says edges are absent. It tests whether learning from model behavior adds anything over the dataset's own structure.
The strongest existing competitor is GRAT. It targets the same failure (early errors poisoning later hops) with tree search. On 2WikiMultiHopQA, GRAT reports 60.6% accuracy for its Llama3-8B baseline, 66.8% after self-training on 11,000 trajectories, and 75.2% with Llama3-70B. Because GRAT spends far more model calls, the fair comparison runs both at matched model size. We report accuracy against total LLM calls and prompt tokens, and give our method GRAT's call budget as an upper cap. Beating GRAT means matching its accuracy within one point at a fraction of its calls, or exceeding the untrained 60.6% figure at 8B without trajectory self-training. Absolute accuracy alone does not count.
Two further comparisons cover other angles. HANRAG is the strongest noise-filtering competitor; on its CompoundMultihop set it reports 71.76% accuracy with 1.24 steps on average (HANRAG), and we will run its relevance discriminator inside our loop as a drop-in replacement for the deletion predictor. For dynamic-retrieval methods, DRAGIN reports 0.3931 F1 on 2WikiMultiHopQA with LLaMA2-13B. We will rerun DRAGIN on the same model rather than compare across model families.
Why not just…?
Why not just always drop the history once the bridge entity is known? This is the real threat. If always-prune matches our method, the context-specific part contributes nothing and the result reduces to "ChainRAG with shorter prompts." We expect the two to tie on most MuSiQue and 2Wiki questions and to separate on ambiguous bridges and on questions whose later hop needs a qualifier from an earlier passage. The measurement phase tells us how large that slice is before we build anything. If it is under a few percent of natural questions, we will report that as a negative finding about how much LLM chains depend on earlier text beyond the extracted entity, and not claim a method.
Why not just filter passages by relevance, as HANRAG does? Relevance to the current sub-question is a different test from redundancy given the resolved bridge. A hop-one passage about Nolan's filmography is relevant to "where was Nolan born" yet adds nothing once "Nolan" is fixed. A hop-one passage naming the release year may look irrelevant yet be what separates two same-named directors. The drop-in comparison inside the same loop, broken down by our ambiguity labels, will show whether the relevance test misses these cases.
Why not just use tree search and let rollouts sort out ambiguity? GRAT does work, with 75.2% at 70B on 2Wiki. But it branches at every node regardless of whether anything is uncertain. Our claim is that most of GRAT's gain on linear-chain failures can be recovered by branching only where the bridge is ambiguous. If GRAT's advantage persists at matched calls, that settles it against us.
Why not just summarize the history with the LLM? Summarization compresses but does not decide what may be forgotten. It also adds a model call per hop and can drop the very qualifier that disambiguation needs. We include an LLM-summary variant at matched token budget as an extra comparison.
Proposed Experiments
The main evaluation uses MuSiQue-Ans and MuSiQue-Full, where unanswerable contrast questions punish chains that guess past a missing link, plus 2WikiMultiHopQA and HotpotQA in the distractor setting. We report answer F1 and exact match, supporting-fact F1 where available, prompt tokens per question, and LLM calls per question. As a reference ceiling, humans reach 78.0 answer F1 on MuSiQue-Ans (MuSiQue); AttrLoRA reports 38.1 exact match there at 7B.
Deletion precision. The first decisive check is deletion precision on held-out questions: the fraction of "deletable" predictions for which the full-history counterfactual gives the same answer. Recall is measured the same way. If precision is not well above always-prune's, the predictor is not detecting context-specific independence.
Ambiguity stress set. The second check is the controlled ambiguity set, where conditional pruning must beat always-prune clearly and approach full-history accuracy.
Distractor robustness. The third check injects distractor passages at hop one. Arc deletion should make later hops insensitive to them whenever the bridge is resolved correctly, while full-history loops degrade. This directly tests the compounding-noise failure noted by HANRAG.
Accuracy per compute. The last check is the accuracy-versus-calls curve against GRAT at 8B and 70B. A win there, together with a measurable slice of truly dependent hops in the measurement study, would support the claim. A tie with always-prune would refute it.
References
- On the Role of Context-Specific Independence in Probabilistic Inference
- Context-Specific Independence in Bayesian Networks
- A New Perspective on Learning Context-Specific Independence
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
- Making Long-Context Language Models Better Multi-Hop Reasoners
- Mitigating Lost-in-Retrieval Problems in Retrieval Augmented Multi-Hop Question Answering
- HANRAG: Heuristic Accurate Noise-resistant Retrieval-Augmented Generation for Multi-hop Question Answering
- GRAT: Guiding Retrieval-Augmented Reasoning through Process Rewards Tree Search
- Selective Induction Heads: How Transformers Select Causal Structures in Context
- ♫ MuSiQue: Multihop Questions via Single-hop Question Composition
- Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps
- DRAGIN: Dynamic Retrieval Augmented Generation based on the Real-time Information Needs of Large Language Models
- HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering