Hallucinations in multi-turn chat often snowball. Once a model states something false in turn three, it treats that claim as established context, and later turns build on it. The difficulty for detection is that these later turns are faithful to the conversation. They agree perfectly with what was said before; what was said before is simply wrong. A judge that reads the history as one flat block of text and asks "is this reply consistent with the context?" will pass them. HalluDial reports that LLMs are more consistent with their own earlier incorrect responses than with gold responses, which is exactly this failure (HalluDial). Even strong models score only 50.14 detection F1 (GPT-4) on DiaHalu's multi-turn dialogues (DiaHalu).
We propose to treat the conversation history as a graph whose nodes are turns, each carrying its text, and whose edges record which earlier turns a later turn depends on. A hallucination score is then passed along those edges. A turn's final score depends on two things: whether its own text looks wrong, and how suspect the turns it rests on are. In two sentences, the idea is this. A reply that faithfully repeats a hallucinated earlier turn should inherit that turn's suspicion through the dependency edge, instead of being judged in isolation. The same propagated scores can then cut suspect turns out of the context before the next reply is generated, which breaks the snowball.
Core Claim
The claim is narrow and testable. On dialogue turns whose error is inherited (the turn is consistent with the history but depends on an earlier hallucinated turn), a judge that propagates soft suspicion scores along turn-dependency edges will beat two alternatives. The first is a strong fine-tuned judge reading linear history. The second is the simpler "judge each turn, hard-mask the flagged ones, re-judge" pipeline. The judge should also stay competitive on ordinary non-inherited hallucinations.
The mechanism fits in one equation. Let $x_t$ be the text of turn $t$ and $\mathcal{N}(t)$ its earlier dependency neighbours. Let $w_{jt}$ be a learned weight on the edge from turn $j$ to turn $t$, and $s_j \in [0,1]$ the suspicion score already computed for turn $j$. The score for turn $t$ is
$$s_t = \sigma\Big(f_\theta\big(x_t,\ \textstyle\sum_{j \in \mathcal{N}(t)} w_{jt}\,[\,e_j ;\ s_j\,]\big)\Big),$$
where $e_j$ is the contextual embedding of turn $j$, $f_\theta$ is the turn encoder with a graph-aggregation layer, and $\sigma$ is the logistic function. Because scores are computed in turn order, suspicion flows forward along dependency edges and never backward.
Relation to Prior Work
Several papers motivate this proposal by documenting the gap. DiaHalu and HalluDial build multi-turn hallucination benchmarks and separate factuality errors (contradicting world knowledge) from faithfulness errors (contradicting the conversation). Both find faithfulness much harder to detect. Both evaluators read raw linear history. MM-Snowball studies six-turn trajectories with an injected error at turn 3 and reports accuracy collapsing below 2% in turns 3–5 for LLaVA-1.5-7B (MM-Snowball). Its fix steers hidden states toward the image, which has no analogue in text-only chat, and it does not model dialogue structure. We borrow its adversarial-bifurcation protocol for building data, but not its mitigation.
Other work supplies the graph machinery. DynaEval builds a directed graph over turns with nine relation types, defined by same versus different speaker and by relative distance, and runs a graph convolution over frozen RoBERTa features to score coherence (DynaEval). We reuse its relation scheme as the fixed backbone of our edges. We differ in three ways: the target is turn-level hallucination rather than dialogue-level quality, a hallucination score travels along the edges, and the text encoder is not frozen. Coreference-aware summarization showed that explicit cross-turn entity links cut wrong-reference errors by 45% in BART summaries (Coreference-Aware Dialogue Summarization). That paper also warns that external coreference parsers break on informal dialogue. This constrains our design: we avoid parser-built edges and learn edge weights over cheap candidate edges instead.
For combining text and structure, we draw on the text-attributed-graph literature, meaning graphs whose nodes carry free text. GraphFormers interleaves neighbour aggregation between Transformer layers and beats cascaded "encode text once, then run a graph network" pipelines by 2.9–6.5% (GraphFormers). GLEM instead alternates training of a language model and a graph network, each producing pseudo-labels for the other. The result is that the language model absorbs structural knowledge and works even when edges are missing at test time (GLEM).
Both were tested only on static citation and product graphs. Dialogue graphs are tiny, with tens of turns, so the scaling objection GLEM raises against layerwise fusion does not apply. We therefore use GraphFormers-style fusion as the main model. We use GLEM-style distillation for a different purpose: to ask whether a plain language-model judge can absorb the propagation behaviour and drop the graph at inference.
Two systems are close but differ in what they ground against. G-Retriever encodes retrieved knowledge-graph subgraphs as soft prompts and cuts hallucinated graph elements by 54% (G-Retriever). Neural Path Hunter corrects hallucinated entities in single responses against a k-hop external knowledge-graph neighbourhood (Neural Path Hunter). In both, the graph is external knowledge and each turn is judged alone. Our graph is the conversation itself, and the quantity of interest is how errors pass between turns. CMT-RAG stores conversation memory as a graph of sub-question reasoning traces. It notes that one wrong early decomposition corrupts everything downstream, but it offers no mechanism for downstream nodes to notice (CMT-RAG). Our propagated suspicion is such a mechanism.
Research Program
We would build three components.
The first is a dependency graph over turns. Candidate edges come from DynaEval's speaker/distance relations inside a window, plus edges between turns that share entity strings or have high embedding similarity. No coreference or dialogue-act parser is used. The model learns a weight on each candidate edge. It is supervised only by the hallucination labels, plus a small auxiliary loss on turns where we know the true dependency.
The second is the judge. It is a Llama-2-7B encoder, the same backbone as HalluJudge, with a graph-aggregation step inserted into its upper layers in the GraphFormers style. It computes $s_t$ turn by turn as in the equation above, and emits both a turn label and the edge weights. The edge weights serve as an explanation: this turn is suspect because it rests on that one.
The third is a generation-time circuit breaker. Before producing turn $t+1$, the dialogue model's context is edited. Turns with high $s_j$ are replaced by a short neutral marker ("[earlier claim withdrawn]") rather than deleted, so the dialogue stays coherent.
Evaluation needs data with known inheritance, which existing benchmarks do not label. We would construct it following MM-Snowball's protocol, but in text. Starting from clean knowledge-grounded and task-oriented dialogues, we inject a false claim at turn 3. We then let an open model continue for four more turns and label each later turn as inheriting the error, independently erroneous, or clean. A human-verified subset calibrates the automatic labels. We call this the inherited-hallucination split and release it alongside results on the unmodified HalluDial and DiaHalu test sets.
Baselines
The naive reference point is zero-shot GPT-4o, prompted with the full linear history to label the final turn. HalluDial reports 78.84 Macro F1 for it, and DiaHalu reports GPT-4 at 50.14 F1. We rerun both on all splits, including the new one.
The strongest existing method is HalluJudge. It is Llama-2-7B fine-tuned on HalluDial's linear dialogues and reports 84.92 Macro F1 on HalluDial (HalluDial). It is the strongest competitor because it has the same backbone, the same training data and the same label space; the only difference is that it lacks graph structure. We retrain it ourselves with the same data, the same number of optimizer steps, and the same hyperparameter search budget as our model. On the inherited split, it additionally gets the same injected-dialogue training data. Beating it means two things. On the inherited split, our model must win by a margin outside the seed-to-seed spread over at least three seeds. On standard HalluDial, it must not fall below HalluJudge's reproduced score. No reported result exists on Lacuna for any method on an inherited-hallucination split, so the reproduced HalluJudge number is the reference.
The key-ingredient baseline is sequential hard masking. We run HalluJudge on each turn in order, replace every turn it flags with the withdrawal marker, and then judge the next turn. It uses the same judge and the same information, but makes a hard yes/no decision per turn and has no learned edges. A cascaded variant freezes Llama-2-7B turn embeddings and runs a DynaEval-style graph convolution with suspicion propagation. This tests whether fusing the graph into the encoder matters or whether a cheap graph on top suffices.
Three ablations isolate propagation itself. The first keeps the graph but removes $s_j$ from the aggregated message, so structure is present but no suspicion flows. The second shuffles edges within each dialogue. The third keeps only temporal-adjacency edges. For mitigation, the baselines are an untouched context and a context pruned by sequential hard masking.
Why not just…?
Why not just fine-tune a long-context judge on the injected dialogues? This is the most serious threat. Given enough examples of inherited errors, a linear judge might learn to look back and doubt earlier turns without any graph. The HalluDial self-consistency finding suggests linear models are biased the other way, but that finding concerns untrained models. So the retrained-HalluJudge baseline gets exactly our training data. If it matches our judge on the inherited split, the graph adds nothing, and we would report that. The GLEM-style distilled judge sits between the two: a win for it would mean structure matters at training time but not at inference.
Why not just judge turn by turn and mask what gets flagged? This is the hard-masking baseline. It fails in one predictable place: the injected turn-3 claim is plausible and gets a middling score below threshold, so it is never masked and everything downstream passes. Soft propagation can still add up several middling signals along a dependency chain. The decisive comparison is therefore stratified by how confidently the injected turn itself was scored. If hard masking matches us even on low-confidence injections, we would drop the learned propagation.
Why not just retrieve the relevant earlier turns with dense search and let the judge read those? Retrieval shortens context, but it returns the hallucinated turn as supporting evidence, which makes the inherited turn look better grounded. CMT-RAG reports that flat dense retrieval trails structured trace retrieval by 6.07 exact match on multi-turn multi-hop dialogue. We include a retrieve-then-judge variant on the inherited split to test this directly.
Proposed Experiments
The decisive experiment is Macro F1 on the inherited-hallucination split, compared against retrained HalluJudge and sequential hard masking, with a breakdown by the distance between the injected turn and the judged turn. The prediction is that the gap over the linear judge grows with distance and with the number of intermediate turns that repeat the error, and vanishes for independent errors.
Secondary checks cover four things. We report the standard HalluDial and DiaHalu test sets, broken down by DiaHalu's five categories; propagation should help most on Overreliance and Incoherence and least on Non-factual. We measure agreement between learned edge weights and the known injection dependency. We evaluate mitigation by running the circuit breaker on held-out injected dialogues and measuring the fraction of turns 4–7 that repeat the false claim, against untouched and hard-masked contexts. Finally, we measure added latency per turn, which must stay small for the circuit breaker to be usable online.
A null result is still informative. If the shuffled-edge and no-$s_j$ ablations match the full model, then any gain comes from extra capacity rather than propagation, and the core claim is false.
References
- HalluDial: A Large-Scale Benchmark for Automatic Dialogue-Level Hallucination Evaluation
- DiaHalu: A Dialogue-level Hallucination Evaluation Benchmark for Large Language Models
- MM-Snowball: Evaluating and Mitigating Hallucination Snowballing in Multimodal Multi-Turn Dialogue
- DynaEval: Unifying Turn and Dialogue Level Evaluation
- Coreference-Aware Dialogue Summarization
- GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph
- Learning on Large-scale Text-attributed Graphs via Variational Inference
- G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering
- Neural Path Hunter: Reducing Hallucination in Dialogue Systems via Path Grounding
- CMT-RAG: Complementary Memory Traces for Multi-turn Multi-hop RAG