Systems that answer multi-step questions with retrieval increasingly plan first. They break the question into sub-questions, look up evidence for each, and revise the plan as facts arrive. This guards against the classic failure of step-by-step reasoning, where one early mistake ruins everything after it. It leaves a quieter failure in place. A retrieved passage that is irrelevant but plausible can slip a wrong "fact" into hop one. Every later hop then builds on it, and each local check passes, because each step is consistent with the step before it. What is wrong is the set of accepted facts taken together, and nobody checks the set. Worse, the evidence suggests that language models are bad at exactly this check once the set grows. Adding neutral, true statements to a contradictory collection hides the contradiction even from GPT-4o and o3-mini, which Introducing Verification Task of Set Consistency with Set-Consistency Energy Networks calls the "dampening effect."
We propose to measure and break these distraction cascades. The planner never reads raw retrieved passages. It reads only a compact ledger of extracted facts. A small trained set-level verifier, an energy model that gives one scalar "incoherence" score to a whole collection of statements, scores that ledger after every hop. When the score rises, the system finds the fact most responsible and re-plans only from that hop onward. The claim is testable in two ways. Distractors planted at hop one should raise final error by more than their single-hop effect alone predicts. A ledger-level energy check should catch these cascades as trajectories lengthen, where an LLM asked to check the same ledger increasingly fails.
Core Claim
Two research lines meet here without engaging each other. Work on contextual noise, such as the distraction-aware answer selection in Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering and the preference-trained joint extract-and-answer model in Aligning Extraction and Generation for Robust Retrieval-Augmented Generation, studies one prompt stuffed with up to 25–100 noisy passages. Neither asks how noise moves across hops. Planning systems do work across hops, but they verify locally. Search-in-the-Chain checks each node of its query tree against retrieved documents and backtracks on conflict. REAP tags each extracted fact as a direct answer, a partial clue, or a failure, and re-plans on that tag. LevelRAG summarizes passages and asks whether the accumulated facts suffice. None of the three asks whether the accepted facts are mutually compatible.
Our claim has three parts. First, distraction compounds across hops, and the planner's exposure to raw evidence is the main channel for it. Second, the compounding shows up as emergent inconsistency in the fact ledger, meaning contradictions that appear only when three or more facts are read together. Third, a set-level energy verifier detects these contradictions, and an LLM checker does not, because the ledger grows with neutral bridging facts that dampen the signal.
The closest prior work is SC-Energy itself. It was evaluated only on static statement sets (Set-SNLI, Set-LConVQA, WIQA) and never used inside an iterative pipeline. We reuse its method and move it into a live planning loop, where its job is to decide when and where to re-plan.
This also resolves a design disagreement in the literature. PEARL withholds the document entirely while planning, to avoid distraction. REAP feeds intermediate evidence back into the planner. We take a middle position and test it: the planner sees evidence, but only as distilled, atomic fact statements, never raw passages.
Research Program
The system has three parts. The planner and fact extractor follow REAP's design. The extractor returns structured fact statements with fulfillment levels. The ledger $S_k = \{s_1, \dots, s_m\}$ is the set of fact statements accepted after hop $k$. Here $s_i$ is a single atomic statement such as "X was born in Y." The planner conditions only on the question, its current plan, and $S_k$. That restriction is the "firewall."
After each hop, a verifier $E$ assigns an energy $E(S_k)$ to the ledger, where higher means less coherent. If the energy crosses a threshold calibrated on held-out data, we localize the culprit by leave-one-out, as SC-Energy does:
$$ i^{*} = \arg\min_{i} E(S_k \setminus \{s_i\}). $$
Here $i^{*}$ is the fact whose removal most restores coherence. The planner then discards $s_{i^*}$ along with everything derived from it, and re-plans from the hop that produced it. This is targeted backtracking driven by a set-level signal rather than by a local tag.
Ledgers of short atomic facts fit within the RoBERTa-base input limit noted for SC-Energy. That limit is one more reason to keep raw passages out.
The verifier must be adapted from its original domains to Wikipedia multi-hop facts. MuSiQue is useful here because each question is composed from single-hop questions joined by bridge entities (MuSiQue). From its gold decompositions we build coherent ledgers. We then create emergent inconsistencies by swapping a bridge entity at one hop for a plausible distractor entity. Every pair of facts stays locally plausible, but the chain no longer closes. We also pad ledgers with true but irrelevant facts to train against dampening. The margin-based contrastive training follows SC-Energy. We will report transfer to 2WikiMultiHopQA and HotpotQA without retraining on them.
The second deliverable is a measurement protocol for cascades. At hop $k$ we inject a controlled distractor passage of one of three types. Type one is topically related but answer-free. Type two contains a plausible wrong bridge entity. Type three contains a correct-looking but outdated or conflicting attribute. We then define cascade amplification as
$$ A_k = \frac{\Pr(\text{final error} \mid \text{distractor at hop } k)}{\Pr(\text{hop-}k\text{ error} \mid \text{distractor at hop } k)}. $$
$A_k$ compares how often a planted distractor ruins the final answer with how often it corrupts the step where it was planted. A value well above one at early hops means errors are being amplified downstream rather than absorbed. We will measure $A_k$ under three conditions: the planner reads raw passages, the planner reads only the ledger, and the planner reads the ledger with energy-triggered re-planning.
Baselines
The naive reference is single-pass retrieve-then-read: top-$k$ passages from the same retriever, then a chain-of-thought answer from the same LLM. It shows how much any planning buys. Its known weakness on these benchmarks is documented by the drop from 64.8 F1 on HotpotQA to 32.0 on MuSiQue for a single-paragraph model reported in MuSiQue.
The strongest existing method is REAP, the best reported planner in the file. It reports 68.0 F1 on HotpotQA against R1-Searcher's 63.4, 38.3 F1 on MuSiQue against 33.8 for the runner-up, and 79.6 F1 on 2WikiMultiHopQA. It is the right competitor for two reasons. It already handles partial and noisy evidence through fulfillment levels, and our system is built on its planner and extractor, so any gain is attributable to what we add. We will rerun it rather than copy its numbers. The rerun uses the same e5-large-v2 retriever, the same model pair (a small plan-updating model and a larger extractor), the same corpus, and an equal cap on LLM calls per question. Beating REAP means a gain in F1 on MuSiQue, the benchmark with the fewest shortcuts, that holds across three seeds. It also means a clearly lower $A_1$ under injected distractors, at no more than about 1.3 times REAP's call count. A clean-benchmark F1 gain without a cascade reduction would not support the claim.
Search-in-the-Chain is the representative node-level verifier with backtracking. It reports 56.91% cover-EM on HotpotQA, and we will run it under the same retriever and call budget.
Four ablations isolate the key ingredients. The first replaces the energy verifier with the same LLM prompted to check the whole ledger for contradictions. This isolates "trained set-level model" from "any global check." The second replaces it with a pairwise natural-language-inference checker run over all fact pairs. This isolates "set-level" from "pairwise." The third removes the firewall, so the planner sees raw passages while the verifier stays. The fourth keeps the firewall but drops verification, which measures the firewall alone.
Why not just…?
Why not just have the LLM re-read all its accepted facts at the end and check them? This is our first ablation, and it is the most natural objection. On static sets, the SC-Energy paper found GPT-4o at 0.713 Macro-F1 on Set-SNLI against 0.941 for the energy model, and LLM self-checking at 59.9% on WIQA against 68.7%. The paper attributes this gap to dampening. That evidence comes from static, non-Wikipedia sets, so it does not settle the question for live ledgers. Our test is to plot detection rate against ledger size for both checkers. If the LLM checker's accuracy holds as ledgers grow, the energy model is unnecessary, and we would report the simpler system.
Why not just denoise every hop with a strong filter or a joint extract-and-answer reader? This is the real threat. If each hop's evidence is cleaned well, perhaps nothing cascades. Ext2Gen shows that joint extraction beats separate compressors under heavy noise (0.557 against 0.473 on Llama3.1-8b-instruct). LevelRAG already summarizes per hop. Local filtering, however, judges relevance to the current sub-question. Our type-two distractors are relevant to the sub-question by construction and become wrong only against the rest of the chain. We will run REAP with a per-hop Ext2Gen-style filter as an additional baseline. If it matches our system on type-two injections, the set-level component is not earning its cost.
Why not just sample several trajectories and vote? Voting helps when errors are independent across samples. A persuasive distractor returned by a fixed retriever is seen by every sample, so the errors are correlated. We will include self-consistency voting at a matched call budget. A strong showing would mean the cascade is mostly a sampling artifact rather than a retrieval one.
Why not just hide documents from the planner, as PEARL does? PEARL's plan is static and has no re-planning loop. REAP's gains suggest that intermediate clues help planning. The firewall-only ablation tests the middle position, and the raw-passage ablation tests the opposite extreme.
Proposed Experiments
The main evaluation runs on HotpotQA (full-wiki), MuSiQue-Ans, and 2WikiMultiHopQA. Each is run in two forms: as released, and with the controlled distractor injections at hops one through three. The metrics are answer F1, the amplification $A_k$, detection rate and leave-one-out culprit accuracy against the known injected fact, and calls and latency per question.
Three checks decide the project. The first asks whether cascades exist: $A_1$ must exceed one meaningfully for REAP with raw-passage planning. If it does not, the premise fails and we stop. The second is the dampening test. LLM-checker detection should fall as padding facts are added to ledgers, while energy detection should stay roughly flat. If both stay flat, we drop the trained verifier. The third measures transfer: whether energy-triggered re-planning lowers $A_1$ and raises MuSiQue F1 over REAP and over the per-hop-filter baseline at matched budget. A secondary analysis on UDA FinHybrid, where long-context GPT-4 scored 0.4 against 45.9 with retrieval, will probe whether ledger checks extend to numeric facts. We expect the verifier will need numeric-specific contrastive examples there, and we will report it as an exploratory result.
References
- Introducing Verification Task of Set Consistency with Set-Consistency Energy Networks
- Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering
- Aligning Extraction and Generation for Robust Retrieval-Augmented Generation
- Search-in-the-Chain: Interactively Enhancing Large Language Models with Search for Knowledge-intensive Tasks
- REAP: Enhancing RAG with Recursive Evaluation and Adaptive Planning for Multi-Hop Question Answering
- LevelRAG: Enhancing Retrieval-Augmented Generation with Multi-hop Logic Planning over Rewriting Augmented Searchers
- PEARL: Prompting Large Language Models to Plan and Execute Actions Over Long Documents
- ♫ MuSiQue: Multihop Questions via Single-hop Question Composition
- UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-World Document Analysis