Inherited Hallucinations in Multi-Turn Multi-Hop RAG

When a chatbot answers a chain of related questions across a conversation, it keeps reusing facts it stated earlier. Suppose it said in turn 3 that the director of a film was a particular person and was wrong. By turn 7 it may confidently answer a follow-up about that person's birthplace. The final answer is wrong, but nothing went wrong in turn 7's retrieval. The error was inherited from the conversation's own history. We propose to measure this failure directly and to fix it with a small mechanism. For every intermediate fact the assistant asserts, it records where that fact came from. It refuses to build on any earlier assertion that was never checked against a read document. When it cannot check, it says so to the user instead of asserting.

This matters because the two communities that could catch the failure each look at the wrong thing. Multi-hop retrieval-augmented generation (RAG), where a language model answers by first searching a document collection, sometimes over several dependent lookups, scores only the final answer. Dialogue evaluation scores whole turns for qualities like truthfulness or coherence without access to the evidence chain. Neither can tell an inherited error from a fresh retrieval failure, and the remedies for the two are different. Our core claim has two parts. First, a meaningful share of wrong answers in multi-turn multi-hop dialogue are inherited errors. Second, a per-hop, evidence-anchored assessment exposes them where holistic judges and final-answer accuracy do not. We also test a third, weaker claim: a provenance gate plus honest hedging reduces them at much lower cost than re-verifying everything every turn.

What the literature already covers, and what it does not

The gap is motivated by three findings. Interleaving retrieval with step-by-step reasoning, as in IRCoT, cuts reasoning-chain factual errors by up to half in single-turn QA. That shows intermediate hops are where hallucinations enter. Holt et al. describe autoregressive compounding: a wrong token becomes context that later steps treat as fact. Our hypothesis is that dialogue history plays the same role one level up. Roh and Han's READ-GATE study shows that agents often answer from search snippets without reading documents. A simple rule that rejects finishing an answer with zero document reads recovers 14.9 to 19.9 points on those failed trajectories. It also shows that giving a model more hidden thinking makes this skipping worse.

Their gate, however, only checks the current episode. It cannot see that a bridge entity pulled from turn 3 was itself never read. We generalize the READ-GATE rule from "did you read anything" to "is every fact you are building on traceable to a read." The closest system is CMT-RAG. It keeps a session-level graph of sub-question traces and paragraph IDs for multi-turn multi-hop dialogue. It reports 41.73 exact match (EM) on the long-dialogue split of MuMu-QA, a 6.07-point gain over direct RAG, with per-turn latency falling from 1.61s to 0.83s.

CMT-RAG stores traces to improve retrieval. It does not verify them before reuse, does not distinguish asserted from hedged facts, and evaluates only EM and latency. We differ on those three points: provenance-gated reuse, surfacing uncertainty as a dialogue act, and per-hop evaluation.

On the evaluation side, AMULET structures LLM judges around dialogue acts (the communicative function of a turn, such as asserting, asking, or correcting) and Gricean maxims, including Quality, meaning truthfulness. It finds that 78% of preference pairs are distinguishable this way. Its Quality judgment is made without any evidence corpus, and the authors themselves note that judges fabricate rationales. DynaEval models dialogue as a graph and reaches 0.482 Spearman correlation on FED, but it measures coherence rather than grounding. CORAL adds citation precision to multi-turn RAG evaluation; its best history strategy, LLM summarization, reaches only 31.1%. CORAL's conversations follow topic drift rather than explicit dependency chains, so it cannot attribute an error to a specific hop. Our assessment borrows AMULET's dialogue-act lens and CORAL's citation checks. It then anchors both to gold hop structure.

Core Claim and Measurement

The key enabler is that MuSiQue builds each multi-hop question by composing single-hop questions with known answers and supporting paragraphs. MuMu-QA, which spreads MuSiQue sub-questions across dialogue turns, inherits that gold structure. For every assistant turn we can therefore decompose the response into claims about specific hops. Each claim is labeled correct or incorrect against the gold sub-answer and grounded or ungrounded according to whether a read document supports it. Each claim is also tagged with the dialogue act used to convey it: flat assertion, hedge, clarifying question, or correction.

Three quantities follow from these labels. The first is hop faithfulness, the fraction of asserted hop claims that are correct and grounded. The second is assertion calibration, the gap between groundedness of asserted versus hedged claims; a well-behaved assistant hedges exactly what it could not verify. The third, the central one, is the inherited-error rate:

$$ \text{IER} = \frac{\#\{\text{wrong final answers whose first wrong hop was asserted in an earlier turn}\}}{\#\{\text{wrong final answers}\}}. $$

Here the "first wrong hop" is the earliest hop in the gold dependency chain whose claimed value differs from the gold sub-answer.

The first deliverable is a diagnostic study that computes IER for existing systems. If IER is small, say under 10% for all systems, the inheritance hypothesis is wrong. We would then report that the multi-turn problem reduces to per-turn retrieval quality, which is itself a useful result. Friction is scored alongside: the number of clarifying turns and the added latency. This keeps a system from winning by hedging everything, which is the helpfulness-versus-caution trade-off that Ou et al. warn about.

Research Program

The intervention is a provenance ledger. Each time the assistant commits to an intermediate entity, the ledger stores the entity, the hop it answers, the document ID it was read from (or a null marker if it came from a snippet, from parametric memory, or from the user), and the dialogue act used. Before a later turn may use a ledger entry as a retrieval key or reasoning premise, a gate checks for a read-document pointer. If the pointer is absent, the system re-verifies that single hop with one targeted retrieval-and-read. If verification fails, the system switches its dialogue act: it states the dependency as uncertain or asks the user, rather than asserting.

This design differs from full re-verification. Only unanchored entries are re-checked, so cost grows with the number of ungrounded commitments rather than with the length of the dialogue. It also differs from CMT-RAG's trace graph, which is used for retrieval but never audited.

We add a stress condition that no existing benchmark has: user-introduced false premises. From MuSiQue gold chains we write follow-up turns where the user asserts a wrong bridge entity ("since X directed it, where was X born?"). These test whether the system inherits errors from the user as readily as from itself. MultiHop-RAG's null queries and MuSiQue-Full's unanswerable contrast pairs supply hops where the correct behavior is to decline.

Baselines

The naive reference point is direct RAG. It retrieves once per turn using the current user message concatenated with the previous turn, and it anchors how bad things are without any history handling.

The first strong family is history compression from CORAL. Conversational query rewriting and LLM summarization are run with the same retriever and generator. We use them because the file reports summarization as the best citation-precision strategy in multi-turn RAG (31.1%).

The second is IRCoT applied within each turn. It is the standard single-turn multi-hop method and isolates whether per-turn iterative retrieval alone eliminates inherited errors.

The strongest existing method for this exact setting is CMT-RAG, at 41.73 EM on the MuMu-QA long-dialogue split as reported in CMT-RAG. If its code is unavailable we will reimplement its trace-graph memory on the same backbone and label the reimplementation clearly. Only matched-backbone numbers will be compared against ours, with the published 41.73 as a reference. Beating it means higher EM on the long split at no more than a fixed multiple of its retrieval calls per turn, together with a lower IER.

We also run per-turn READ-GATE, which enforces a read before finalizing each turn. This tests whether the existing gate already captures the benefit.

The most dangerous baseline is exhaustive re-verification: every hop of the dependency chain is re-retrieved and re-read on every turn. It should nearly eliminate inherited errors at high cost. The fair comparison with it is accuracy at matched retrieval budget and latency.

Two ablations isolate our ingredients. The first keeps the ledger but removes the gate, so provenance is recorded but not enforced. The second keeps the gate but removes the dialogue-act switch, so failed verification leads to a best guess rather than a hedge.

All systems share one generator (one 7B-class open model and one API model), one retriever and index, identical prompts where applicable, and the same small tuning budget on a held-out MuMu-QA development split.

For the evaluation side, the baselines are a single 1-to-5 LLM judge, an AMULET-MAXIM-style Quality judge, and CORAL citation precision. Each is scored on how well it predicts our gold per-hop correctness labels. AMULET's reported 67.0% versus 66.1% on HH-TEST is a preference-accuracy figure on a different task and serves only as context. We found no reported result on Lacuna for any judge's agreement with hop-level gold labels.

Why not just…?

Why not just re-verify every hop on every turn? This is a real threat. On accuracy alone it may match or beat the ledger. The proposal stands only if the gated ledger comes close in accuracy while using substantially fewer retrieval calls and less latency. CMT-RAG's latency numbers suggest per-turn cost matters in dialogue, but that is an argument, not evidence. We will plot EM and IER against retrieval calls for both. If exhaustive re-verification dominates at every budget, we will drop the intervention and present the paper as a measurement contribution.

Why not just apply READ-GATE at each turn? READ-GATE checks that the current episode read something. It does not check whether a premise carried over from history was ever grounded. So we expect it to lower fresh-retrieval errors and leave IER largely unchanged. The per-turn READ-GATE baseline tests this directly. If it also collapses IER, the ledger adds nothing, and we will report that.

Why not just ask a strong LLM judge whether each turn is truthful? AMULET shows structured judges beat vanilla ones, but its Quality judgment has no evidence access. A judge that cannot see which hop the answer depends on will score a fluent answer built on an inherited error as plausible. We measure this by computing each judge's agreement with gold hop labels, separately on inherited-error and fresh-error turns. The prediction is that agreement drops sharply on inherited errors. If a judge given the retrieved documents matches our hop-level labels, the cheaper judge suffices for evaluation, and we would say so.

Why not just use a bigger model? MultiHop-RAG reports GPT-4 at 0.56 accuracy with retrieved chunks versus 0.89 with gold evidence, so scale does not remove the retrieval bottleneck. READ-GATE finds more hidden thinking increases evidence skipping. Running both a 7B model and an API model lets us check whether IER shrinks with scale.

Proposed Experiments and Decisive Checks

The first decisive check is the diagnostic. We compute IER for all baselines on MuMu-QA and on the false-premise variant. The hypothesis survives if IER is substantial for at least CMT-RAG and summarization-based RAG, and if it is higher on long dialogues than short ones.

The second check is the evaluation claim. On a sample of about 500 turns, two annotators label hop correctness, groundedness, and dialogue act, which lets us validate the automatic labels. We then test whether holistic judges and citation precision fail specifically on inherited errors.

The third check is the intervention. Relative to CMT-RAG on the same backbone, the gated ledger should lower IER and raise long-split EM at matched retrieval budget. It should improve assertion calibration without inflating clarifying turns beyond a preset friction cap. Its false-premise acceptance rate should also be lower. The two ablations then say whether any gain comes from enforcing provenance or from hedging honestly.

MultiHop-RAG null queries and MuSiQue-Full unanswerable pairs check that the gate turns unverifiable hops into declines rather than fabrications. Results on HotpotQA and 2WikiMultihopQA, with dialogues built from their gold supporting facts, check that the effect is not specific to MuSiQue's construction.