When a search system is unsure what a user wants, a common fix is to have a language model write extra material to search with: a guessed answer, a few related terms, or a short made-up passage. This is called query expansion. In a single query this helps on average, but in specialist domains the model often guesses wrong, and the wrong guess pulls retrieval away from what the user meant. In a conversation the problem gets worse. The guessed passage, or the documents it retrieved, usually stays in the dialogue history. Every later turn is then rewritten and expanded while conditioning on it. We propose to measure this carry-over effect directly and to test a simple remedy. Machine-generated expansion text is used only for retrieval at the turn that produced it and is never written into the conversation state. Before it is used, each generated sentence is checked against passages retrieved for the user's own words.
The opportunity is concrete. Filtering hallucinated expansion text already works in single queries. GOLFer lets an 8B model match GPT-4o-based expansion on SciFact (nDCG@10 69.4 vs 69.2 for GPT-4o with HyDE). Aligned Query Expansion trains a model to produce useful expansions in one pass, raising NQ top-1 retrieval accuracy from 22.1% unexpanded to 30.8%. Both treat each query as isolated. Multi-turn work shows that errors persist across turns once they enter context. Two examples are the late-turn retrieval failures in RECOR and the fact that models cannot "un-see" retracted content in ChronoMem. No one has asked whether expansion itself is a source of those persistent errors.
Core Claim
Our claim is that in multi-turn retrieval over specialist domains, much of the damage from LLM query expansion comes through persistence rather than through the turn where the bad expansion was generated. A wrong hypothetical passage at turn $t$ lowers retrieval quality at turns $t+1, t+2, \dots$ because later rewrites and expansions condition on it. We further predict that this carry-over harm grows as the domain gets less familiar to the model, since hypothetical text is least accurate where the model lacks domain knowledge.
Two cheap interventions should then recover most of the loss. The first is quarantine: expansion text and the documents retrieved only because of it are never stored in the conversation state. The second is anchoring: a black-box check keeps a generated sentence only if passages retrieved for the unexpanded query do not contradict it.
What is new is not filtering. It is the claim that the unit of harm is the dialogue trajectory, and a design that targets that trajectory without white-box access to model internals.
Research Program
The work has three parts.
Part 1: a carry-over diagnostic. For each conversation we run the same retriever and expander under three history conditions.
- Full carry-over: prior expansions and their retrieved documents stay in history, which is the default in most pipelines.
- Quarantined: history holds only user turns, system answers, and passages actually cited in answers.
- Oracle: gold or human-resolved standalone queries are used.
The quantity of interest is the late-turn retrieval gap between the first two conditions, as a function of distance from the turn where expansion first went wrong.
We also run controlled perturbations, in the spirit of the adversarial trajectories in MM-Snowball, but for text retrieval rather than vision-language dialogue. We plant a contradicted hypothetical passage at turn $t$ and trace recovery over the following turns. This tests whether retrieval shows the mid-dialogue collapse that MM-Snowball reports for generation, where LLaVA-1.5-7B accuracy falls below 2% in turns 3–5.
Part 2: the domain-familiarity axis. RECOR spans 11 domains. We score each domain by the expander's closed-book accuracy on held-out domain questions, as a proxy for the knowledge gap that TheoremQA documents in STEM (open models below 15%). We then test whether carry-over harm is monotone in unfamiliarity.
Part 3: the anchoring check. At each turn we first retrieve the top $k$ passages using only the user's current turn, resolved against quarantined history. We call this the anchor pool. Each sentence of the generated expansion is then scored with the same NLI model GOLFer uses (DeBERTa-v3-large) against the anchor pool, and sentences contradicted by any anchor passage are dropped. We compare a strict variant, which keeps only entailed sentences, with a lenient variant, which keeps anything not contradicted.
This comparison addresses a real tension. Strict anchoring can only confirm what unexpanded retrieval already found, which defeats the purpose of expanding. Lenient anchoring keeps new vocabulary but may keep plausible fabrications.
The check needs one extra retrieval and roughly $k \times s$ NLI calls, where $s$ is the number of expansion sentences. It needs no logits and no multiple samples, so it applies to API models. This addresses the two limitations GOLFer's summary lists: white-box dependence and multi-sample cost.
Baselines
Naive reference. Concatenated history plus the current turn, sent to BM25 and to a dense retriever, with no expansion. It tells us whether expansion helps at all in this setting.
Standard rewriter. An LLM prompted to rewrite the turn as a standalone query, with no expansion. This is the default practice. Query Rewriting & Multi-Turn Entity Graphs shows rewriters drop keywords, which is the reason expansion is attractive.
Per-turn generative expansion. HyDE/Query2Doc-style expansion at every turn with full carry-over. This is the condition we expect to fail.
Strongest expansion competitor. GOLFer's filter applied independently at each turn, using the same LLaMA-3-8B open expander so its white-box signals are available. It is the best existing hallucination filter for expansion. On SciFact it reports nDCG@10 69.4, and on MS MARCO sparse retrieval it reports MRR@10 19.9 against 19.0 for Query2Doc. We run it with full carry-over, since that is how it would be deployed conversationally. We also run it with quarantine, which separates the value of quarantine from the value of filtering.
Strongest no-expansion competitor. The history-plus-explicit-reasoning retrieval reported in RECOR. It raises nDCG@10 from .236 to .479, and its reasoning-specialized retriever DIVER reaches .584. Beating .584 on RECOR is the bar for claiming a practical gain. Because that number may depend on retriever choice, we also compare every condition on the same retriever backbone.
Fusion. Retrieval fusion over unexpanded, expanded, and rewritten queries, as in RALI@TREC iKAT 2024. Its "chorus effect" suppresses single-path noise without any filter.
Fairness and ablations. All systems use the same corpus, the same first-stage retriever and reranker, the same expander model, and matched tuning budgets. Tuning covers $k$ and the NLI threshold for ours, and the number of expansions and fusion weights for the others, on the same development split. Two ablations isolate the key ingredient: quarantine without anchoring, and anchoring with full carry-over.
What counts as winning. We beat per-turn GOLFer if quarantine plus anchoring matches or exceeds it on late-turn nDCG@10, meaning turns 5 onward, with no logit access. The hypothesis is supported if the quarantine-only ablation already closes most of the gap between full carry-over and oracle.
Why not just…?
Why not just drop expansion and use a multi-turn fine-tuned dense retriever? ChatQA shows a multi-turn retriever can match GPT-3.5 query rewriting, and its full system reaches 58.25 average on ChatRAG Bench. This is a real threat. We include it as a baseline on the ChatRAG Bench subsets with specialist content: DoQA, ConvFinQA, and INSCIT. If it beats all expansion conditions in unfamiliar domains, the conclusion is that expansion should be avoided there, and our diagnostic will have shown why.
Why not just run GOLFer at every turn? Per-turn filtering removes errors it detects. But an undetected wrong sentence still enters history under full carry-over, and the filter needs logits. The two-by-two design (GOLFer with and without quarantine) shows directly whether filtering alone leaves a carry-over residue.
Why not just clip or summarize history? Clipping breaks inertia in Mitigating Conversational Inertia, but it discards early constraints. That paper also reports that LLM summaries over-claim and conclude prematurely. CORAL finds summarization gives the best citation precision among its compression strategies, yet that precision is only 31.1%. Quarantine is narrower than either approach: it keeps everything the user said and everything that was cited, and removes only machine guesses. We run clip-context and summary-history as additional history conditions.
Why not just align the expander with retrieval rewards, as AQE does? AQE's reward is the rank of judged relevant documents. RALI@TREC iKAT 2024 shows a correct domain expansion ("DASH diet" to its full name) dropping nDCG@10 from 52.9 to 29.4 because the new documents were never judged. Alignment on pooled conversational judgments would learn to avoid precisely the domain vocabulary that expansion exists to add. It is also trained single-turn. We treat aligned expansion as future work that could sit inside quarantine rather than replace it.
Proposed Experiments and Decisive Checks
Benchmarks. RECOR is the primary benchmark: 707 conversations, 11 domains, and the reported late-turn failure concentration. We use the specialist subsets of ChatRAG Bench, with DoQA in particular for Cooking, Travel, and Movies FAQ dialogues. We use CORAL as an open-domain control where the knowledge gap should be small.
Metrics. - nDCG@10 and Recall@20 per turn index. - The carry-over gap between full-carry-over and quarantined runs. - Downstream answer F1 and citation precision with a fixed generator.
Guarding against pooling bias. Because judgment pools can penalize good expansions, we report the fraction of unjudged documents in the top 10 for every system. We also have annotators judge a stratified sample of unjudged top-ranked documents from each system, so that a drop in nDCG can be checked against real relevance.
Decisive checks. There are three, and each can falsify the proposal.
- Carry-over harm must be present. The full-carry-over versus quarantined gap on RECOR should be positive and should grow with turn distance after the first bad expansion. If the gap is near zero, expansion errors do not persist, and the proposal reduces to per-turn filtering.
- The domain axis must hold. Across RECOR domains, the gap should correlate positively with expander unfamiliarity, with CORAL showing the smallest effect. A flat relationship would undercut the link to the domain knowledge gap.
- Quarantine should do most of the work, and cheaply. Quarantine plus lenient anchoring should match GOLFer-with-quarantine on late turns while using only black-box access. If strict anchoring wins instead, expansion adds little beyond the anchor pool in these domains. That is also a useful negative result about when expansion is worth running at all.
References
- GOLFer: Smaller LMs-Generated Documents Hallucination Filter & Combiner for Query Expansion in Information Retrieval
- Aligned Query Expansion: Efficient Query Expansion for Information Retrieval through LLM Alignment
- RECOR: Reasoning-focused Multi-turn Conversational Retrieval Benchmark
- ChronoMem: Version Control and Semantic Rollback for Large Language Model Agent Memory
- MM-Snowball: Evaluating and Mitigating Hallucination Snowballing in Multimodal Multi-Turn Dialogue
- TheoremQA: A Theorem-driven Question Answering Dataset
- Query Rewriting with Expansion and Multi-Turn Entity Graphs for Answer Selection
- RALI@TREC iKAT 2024: Achieving Personalization via Retrieval Fusion in Conversational Search
- ChatQA: Surpassing GPT-4 on Conversational QA and RAG
- Mitigating Conversational Inertia in Multi-Turn Agents
- CORAL: Benchmarking Multi-turn Conversational Retrieval-Augmented Generation