Treating a Chatbot's Own Earlier Answers as an Untrusted Source

In a long conversation, a retrieval-augmented chatbot reads two kinds of text before each reply. One is retrieved documents. The other is everything said so far, including its own earlier answers. Work on source credibility has taught these systems to doubt documents, by weighing publishers, cross-checking sources and reading reliability metadata. Yet the system's own past turns enter the context with no provenance and full implicit trust. A single invented fact in turn three becomes a premise in turn seven. We propose to treat the assistant's earlier answers as one more source, with its own reliability score, and to stop mixing them with the user's intent.

Concretely, the context is split into two streams. The pragmatic stream holds what the user has been trying to do: their turns plus a short label for each turn's dialogue act, meaning the turn's conversational function, such as a new request, a follow-up, or a correction. The evidence stream holds retrieved passages together with factual claims extracted from the assistant's earlier answers. Every item in it carries a source tag and a credibility score. That score is set by checking the claim against retrieved evidence and is revised by the user's dialogue acts: a correction lowers it, uncontested reuse leaves it unchanged. The bet is that this stops hallucination cascades without losing the follow-up coherence that comes from discarding assistant history altogether.

Core Claim

Multi-turn hallucination cascades are partly a provenance failure. The model cannot tell "the user wants X" from "I previously asserted Y", and it gives Y the same authority as retrieved evidence. If prior assistant claims are moved into an evidence stream and scored like any other source, two things should follow. Injected or self-generated false claims should propagate to later turns far less often. Follow-up turns that legitimately depend on earlier answers should keep the quality they have under full-context conditioning. The second half is what separates this from omitting assistant turns. The design also predicts a third behaviour. When the evidence stream holds a low-credibility claim that conflicts with retrieval, the right move is often a clarifying or repair act ("earlier I said X, but the sources say Y") rather than a confident new answer.

Relation to Prior Work

The direct motivation comes from Do LLMs Benefit From Their Own Words?. That paper shows that keeping prior assistant turns spreads errors. It classifies user turns into acts ("New Ask", "Follow-up with Feedback", "Follow-up without Feedback") and drops assistant history accordingly, reaching over 95% of full-context performance with 30% less context. Its remedy is all-or-nothing omission of whole turns, and it uses no retrieval. We keep the content but demote it, so a follow-up that needs the earlier answer can still use it at a lower trust level.

The dialogue-act side also draws on Navigating Rifts in Human-LLM Grounding. That work found that LLMs rarely clarify or repair, and that an early grounding failure triples the chance the next turn also fails. It is a constraint on our design: we cannot assume the generator will choose a repair act unprompted. That paper's forecaster-triggered prompt raised Llama-3.1-8B from 24.22% to 54.48% on its benchmark, which suggests an explicit trigger works. In our case the trigger comes from the credibility state rather than from ambiguity in the user's request.

The method draws on three lines of work. From retrieval augmented generation with estimation of source reliability (RA-RAG) we take reliability estimation by cross-checking and an alignment check against evidence (AlignScore). RA-RAG is single-turn and scores external sources only; we apply the same logic to the assistant as a source. From Resolving Conflicting Evidence in Automated Fact-Checking we take the finding that showing credibility information to the generator beats pre-filtering. Its background augmentation gave a 10% absolute F1 gain for LLaMA-3.1 on the ModC split of the conflicting-evidence benchmark CONFACT. That supports keeping low-credibility claims visible but marked rather than deleting them. From Focused Attention Improves Document-Grounded Generation we take the stream separation. Its DoHA model gives documents dedicated decoder cross-attention heads, apart from the heads that read chat history. DoHA treats the document as ground truth and concedes it will produce "grounded lies"; we add per-item credibility and route the model's own history into the document side.

Two further papers set the scope. Adaptive Memory Admission Control For LLM Agents decides what enters long-term agent memory; we act within a single session and weight content rather than gate it. RAGTruth shows that fine-tuned span-level detectors beat prompted GPT-4. It supplies both our measurement tool and our strongest filtering baseline.

Research Program

The system has four parts, and each is meant to be replaceable by a simpler version so we can measure its contribution.

The first part is a dialogue-act tagger for user turns. It uses the three-way taxonomy from the omission paper plus a feedback polarity: correction, acceptance, or none. It is run as an instruction-tuned classifier and validated against that paper's labelled WildChat and ShareLM conversations. No reported tagger accuracy was found on Lacuna, so we report our own.

The second part is a claim extractor. After each assistant turn it splits the answer into atomic factual claims, links each to the turn it came from, and scores it with an alignment model against that turn's retrieved passages, as RA-RAG's alignment filter does. The credibility of a claim $j$ at turn $t$ is

$$c_j^{(t)} = \sigma\big(\alpha\, s_j + \beta\, f_j^{(t)} + \gamma\big),$$

where $s_j$ is its alignment score against retrieved evidence, $f_j^{(t)}$ is the cumulative user feedback directed at it ($-1$ per correction act aimed at it, $0$ otherwise), $\sigma$ is the logistic function, and $\alpha, \beta, \gamma$ are fit on a held-out set. Claims without a matching passage get $s_j$ at a neutral prior, so "unverified" is kept distinct from "contradicted".

The third part is the two-stream generator, built at two levels of cost. The prompt-level version keeps a frozen decoder-only model and puts the pragmatic stream and the evidence stream in separate structured blocks. Each evidence item is rendered as a short record (source type, turn, credibility bucket). Assistant prose is removed from the pragmatic stream and replaced by its act label and a one-line move summary. The architectural version fine-tunes an open 8B model with low-rank adapters. Some attention heads attend only to evidence-stream tokens, in the style of DoHA, and a learned credibility embedding is added to each evidence item. Comparing the two tells us whether separation needs to live in the weights.

The fourth part is a repair trigger. When the current answer would rely on an evidence item whose credibility falls below a threshold and conflicts with a retrieved passage, the generator is prompted to open with an explicit repair or clarification act.

Evaluation needs a benchmark that does not yet exist: multi-turn, grounded, with known pollution. We build it from Wizard of Wikipedia and CMU_DoG conversations. In a gold assistant turn $k$ we swap one fact for a plausible false one. We then generate turns $k{+}1$ onward against real follow-up user turns, along with scripted follow-ups that test the planted fact. Three conditions are crossed with this. In the first, retrieval contradicts the planted claim. In the second, retrieval is silent about it. In the third, a later user turn corrects it. We also convert CONFACT HumC claims into two-turn exchanges, where the user follows up on a verdict, to test conflicting external sources and self-generated claims together.

Baselines

The naive reference point is full-context conditioning. The same generator sees the whole history and the retrieved passages concatenated in one flat sequence. It sets the propagation rate we are trying to reduce and the follow-up quality we must not lose.

The strongest existing method for this exact problem is act-guided assistant omission from Do LLMs Benefit From Their Own Words?. Its adaptive variant drops assistant turns according to the user's act and reached over 95% of full-context performance at 30% less context. We reimplement it with the same tagger, generator, retriever and decoding settings. Beating it means two things together: a lower false-claim propagation rate at equal or better quality on follow-ups that depend on earlier answers. Omission should do well on propagation and poorly on those dependent follow-ups, so a win on both is the result that matters.

The strongest filtering competitor is a RAGTruth-style detector. RAGTruth reports that its fine-tuned Llama-2-13B reached 78.7% response-level F1, against 63.4% for prompted GPT-4, and that filtering candidate responses with the detector cut a Llama-2-7B model's hallucination rate by 63.2%. We train an equivalent detector on RAGTruth. We then delete the flagged spans from history, or regenerate flagged turns, before the next turn. This tests whether a good detector applied to a flat context is enough.

The most dangerous baseline is inline credibility tagging. The same extracted claims and credibility scores are written into the flat history next to the assistant text, in the style of source background augmentation. It isolates our key ingredient, because it has the credibility information but no stream separation. All systems get the same retrieval budget, the same generator checkpoint, and the same number of tuning runs for thresholds and prompts on a shared development split.

Ablations of the full system remove one piece at a time: the credibility scores (stream separation alone), the act-driven feedback term ($\beta = 0$), and the repair trigger.

Why not just…?

Why not just drop the assistant's earlier turns, as the omission paper does? It is cheap and already shown to keep most quality. It is probably the hardest baseline on propagation. Its cost falls on "Follow-up without Feedback" turns that refer back to content only the assistant produced, such as "use the second option you gave". Our scripted dependent follow-ups measure exactly this. If omission matches us on those turns, its simplicity wins and the case for keeping demoted claims falls.

Why not just run a strong hallucination detector and delete what it flags? RAGTruth shows such detectors beat prompted frontier models. Still, 52.7% span-level F1 means about half of hallucinated spans are missed or wrongly marked, and deletion is irreversible. A detector also cannot represent "unverified but uncontested", which is the silent-retrieval condition. Our comparison on that condition, and on recovery after later user corrections, settles whether graded and revisable credibility adds anything over binary deletion.

Why not just put the credibility scores inline in the ordinary prompt? This is the real threat. Background augmentation already works when written into a flat prompt, and the worry that longer prompts lose information in the middle is not backed by a measurement in our sources. If inline tagging matches the two-stream system, the honest conclusion is that provenance and credibility do the work and separation does not. We would then drop the architectural variant and report the tagging pipeline as the contribution. Comparing prompt-level separation, architectural separation and inline tagging at matched information is designed to give this answer directly.

Why not just instruct the model to distrust its earlier answers? The rifts benchmark found that frontier models stay at 23% to 27% accuracy on grounding tasks and rarely repair without an external trigger. Instructions without per-claim state leave the model to guess which of its claims are suspect. We include this instruction-only baseline and expect it to cut propagation only in the condition where retrieval contradicts the claim.

Proposed Experiments

The headline measurement is false-claim propagation rate on the polluted Wizard of Wikipedia and CMU_DoG benchmark. It is the fraction of later turns that restate or rely on the planted fact, judged by the RAGTruth-style detector and audited by hand on a sample. It is reported separately for the contradicted, silent and corrected conditions. Next to it we report follow-up quality on dependent and self-contained turns, using pairwise LLM-judge preference against the full-context response with human spot checks. We also report repair-act rate when credibility conflicts with retrieval, and context length per turn.

On WildChat and ShareLM we test that nothing breaks on real traffic. Quality should stay within the 95% of full-context performance that omission reaches. On the multi-turn CONFACT conversion we report verdict accuracy and verdict stability across the follow-up turn, against CoVer's reported 88.4% on human-curated conflicts as a single-turn reference (CoVer).

The decisive check is the pattern across conditions. The idea is supported if the two-stream system lowers propagation compared with full context in all three conditions, and matches omission's propagation while beating it on dependent follow-ups. If inline tagging matches the two-stream system everywhere, the stream mechanism is unnecessary and we say so. If no variant beats omission on dependent follow-ups, then keeping demoted claims buys nothing, and the core claim is wrong.