When a user changes the subject mid-conversation, a chat or search system has to stop using the earlier turns. If it doesn't, the old topic leaks into the new answer. Benchmarks that test this skill, such as TopiOCQA, QReCC and CORAL, are built from Wikipedia. Today's language models were trained on Wikipedia. So a model that "recovers" after a topic switch may not have dropped the stale history at all. It may simply know the answer to the new question from memory, and that memory hides the mistake.
We propose to measure this hiding effect directly. We generate pairs of conversations with identical structure: the same turn plan, the same switch points, and the same kinds of distracting history. One member of each pair is grounded in real encyclopedic facts. The other is grounded in a procedurally generated fictional knowledge base that no model can have memorized. The difference between the two tells us how much of each method's apparent topic-switch robustness comes from recall rather than from managing context. The same comparison shows whether the ranking of history-management methods on static benchmarks can be trusted.
Core Claim
Our claim is that benchmark contamination does not just add a constant offset to conversational scores. It interacts with topic drift. Memorized knowledge covers for two opposite errors. The first is over-carry: answering the new question with facts about the old topic. The second is under-carry: dropping a constraint the user set earlier and still wants applied. Because memory covers for both, contamination should compress the differences between history-management strategies, such as feeding the full history, rewriting the query, or summarizing the history.
We will test this with a difference-in-differences measure. Let $\Delta_{w}$ be a method's accuracy on matched no-switch turns minus its accuracy on switch turns, measured in world $w \in \{\text{real}, \text{fictional}\}$. The masking effect is
$$M = \Delta_{\text{fictional}} - \Delta_{\text{real}}.$$
If $M$ is large and differs across methods, static benchmarks misrank those methods. If $M$ is near zero, contamination is only an additive nuisance and static benchmarks remain valid for comparing methods. That would be a useful negative result, and the design is built so it can come out either way.
Relation to Prior Work
Two strands of prior work motivate the question.
The first shows that history acts as noise after a topic switch. Saving Dense Retriever from Shortcut Dependency shows that retrievers keep 78–90% of their performance even when the current question is deleted. Beyond Continuity introduces an over-carry rate and finds a "continuity bias" that gets worse when the switch happens late in the conversation. Its authors note that they score switch detection and context shortlisting in isolation, not the downstream answers. We score the answers. CORAL finds that summarizing the history gives the best citation precision. Its authors themselves flag Wikipedia contamination as an unaddressed threat to that result. Our design tests that threat directly.
The second strand shows that dynamic generation exposes memorization. DyVal generates reasoning tasks from graphs, and models that are strong on static math benchmarks score near zero on them. INVITE builds fresh false-premise questions from DBpedia triples. Both work on single-turn tasks only. We borrow their method of instantiating tests from a graph and extend it to multi-turn conversations with scheduled topic switches.
We reuse two methods directly. We borrow the decompose-and-verify approach for building dialogues from RECOR. We borrow the conversation-transition strategies (random walks within and across Wikipedia page trees) from CORAL.
One prior result could look like a contradiction. TopiOCQA's own closed-book result, where GPT-3 reached only 31.8 F1 (TopiOCQA), suggests that memory alone does not solve that benchmark. But that measurement used an older model and asked whether memory solves the task. Our question is different: whether memory rescues specific switch-turn failures for current models. That is a narrower and still open question.
Research Program
The generator. We first sample a small knowledge graph of entities, typed attributes and relations. In the real world, the facts come from DBpedia-style triples, as in INVITE. In the fictional world, the facts come from an isomorphic graph with invented names and values. "Isomorphic" means it has the same shape, attribute types and relation types.
Conversation plans. A conversation plan is a walk over the graph with a scheduled sequence of switches. We define five switch types:
- a hard switch to an unrelated entity;
- a cue-bridged switch, which uses a phrase like "switching gears";
- a sibling switch to a closely related entity, as a soft or partial transition;
- a return switch back to an earlier topic;
- a constraint-carrying switch, where a user restriction set before the switch (for example, "only things after 2000") must still apply.
Switch position is varied systematically, which lets us revisit the position bias found by Beyond Continuity.
Adversarial stale answers. Crucially, each switch turn is designed so that the stale topic offers a plausible wrong answer. For example, the new question asks for an attribute that the old entity also has. This lets us score downstream over-carry as a stale-answer rate: how often the model's answer matches the old entity's value. It also lets us score under-carry as a constraint-violation rate on constraint-carrying switches.
Documents and realization. For each entity we render a page in the same templated, paraphrased style in both worlds. This keeps surface form matched between twins. An LLM turns plans into natural utterances. Every utterance is then checked against the graph so that each gold answer is fully determined by it.
Two kinds of noise. We cross the conversation-history noise with retrieval noise in a factorial design. "Horizontal" noise is the stale turns in the conversation. "Vertical" noise is distractor documents among the retrieved passages, and we compare distractors drawn from the old topic's pages against random ones. This lets us separate the two sources of noise that the noise-sensitivity literature and the topic-drift literature have each studied alone.
Validation. Before drawing conclusions, we will check two things. For convergent validity, we expect real-world twins to reproduce the qualitative pattern of TopiOCQA and Beyond Continuity, including high over-carry for open-weight models and its decay with switch position. For naturalness, human raters will compare a sample of our conversations against TopiOCQA dialogues.
Baselines
Naive reference: full history. The do-nothing reference feeds the full, uncompressed history and all retrieved passages to the reader, with an instruction to ignore irrelevant turns. Beyond Continuity and CORAL both show this degrades over long dialogues. It still has to be measured in both worlds, because it is the condition where memory should help most.
Other history-management strategies. We will also run the three strategies CORAL benchmarks: keeping only the last response, conversational query rewriting, and LLM summarization. CORAL reports 31.1% citation precision for summarization, its best method. Summarization is the strongest existing method on this question because it explicitly discards stale history. It is therefore exactly the method whose advantage contamination could inflate, through under-carry being hidden.
Strongest retriever. On the retrieval side, the strongest competitor is the retriever trained with model-based hard negatives from Saving Dense Retriever. That paper reports Recall@10 rising from 50.3 to 63.0 on QReCC switch turns. We retrain it on real-world twin training conversations and evaluate on held-out twins from both worlds.
Noise-robust readers. For retrieval noise we include two methods:
- Ext2Gen, which trains the reader to extract evidence before answering. It reports an average score of 0.557 versus 0.473 for default Llama-3.1-8B-Instruct under heavy retrieval noise. That is single-turn evidence, so its conversational behaviour is untested.
- PRCA, a small reinforcement-learned compressor placed between retriever and reader. It reports an average 9% accuracy gain on TopiOCQA.
Fair comparison. Every method uses the same reader models: Llama 3.1-8B and Gemma 3-12B (open-weight, as in Beyond Continuity), plus GPT-4o and Claude 3.7 Sonnet. Every method gets the same input-token budget, the same retrieved pool, and the same number of tuning runs on a shared real-world development split. Nothing is tuned on fictional conversations.
What counts as success. "Beating" a baseline is not the goal here. The proposal succeeds if two things hold. First, $M$ is significantly nonzero for at least one strong method. Second, the method ranking measured on fictional twins differs from the real-twin ranking. We will report Kendall's tau between the two rankings, with conversation-level bootstrap intervals.
Ablation of the key ingredient. The ablation that isolates our key ingredient, structural pairing, replaces fictional twins with unpaired fictional conversations. If unpaired data gives the same conclusions, pairing is unnecessary.
Why not just…?
Why not just rename entities in TopiOCQA? This is the most serious threat, and we include it as a baseline condition. Renaming keeps natural human dialogue, which our generator cannot fully match. However, swapped names collide with surviving context and with the model's prior knowledge. That creates knowledge conflicts, which are a separate failure mode from memorization. Renaming also gives no control over switch type or position. We will compute $M$ on renamed TopiOCQA too. If it matches our fictional-twin estimate, the cheaper method is sufficient, and we will say so.
Why not just stratify existing benchmark turns by closed-book correctness? Splitting TopiOCQA turns into those a model answers without context and those it does not is cheap, and it yields a correlation. But "answerable from memory" is confounded with how popular an entity is and how easy the question is. Neither can be balanced across switch conditions after the fact. We will run this stratification and compare its implied masking to the twin estimate. Agreement would make the stratified version a useful audit tool.
Why not just use post-cutoff conversations? Post-cutoff data expires as newer models train on newer crawls (Chi et al.). It also cannot be paired with a contaminated twin of identical structure, so it measures a level, not an interaction. DyVal's near-zero scores for statically strong models show the gap a dynamic design can reveal.
Why not just feed everything to a long-context model? Long context is not noise robustness. DAS shows reader accuracy falling as more passages are added. Beyond Continuity shows over-carry persisting in strong models. The full-history condition is our naive baseline, so this alternative is evaluated directly rather than assumed to fail.
Proposed Experiments
Main experiment. We generate about 2,000 twin pairs of roughly 10–14 turns each, comparable in length to TopiOCQA's 13-turn average. We run every baseline under every switch type and switch position, and under both noise factors. The primary outcomes are answer F1 against graph-determined answers, stale-answer rate, constraint-violation rate, and the Beyond Continuity over-carry rate. For retrieval we add Recall@10, and for generation we add citation precision.
Decisive checks. The first decisive check is whether $M$ differs from zero, and whether it differs between summarization and full history. Our prediction is that the advantage of summarization over full history is larger on real twins than on fictional ones. The reason is that memory covers for the constraints summarization drops on real twins, while those drops surface as constraint violations on fictional ones. If instead summarization keeps its advantage on fictional twins with no extra constraint violations, CORAL's conclusion holds up against contamination. Either outcome is informative.
Secondary checks. The second check is whether the position decay found by Beyond Continuity appears equally in both worlds. The third is whether retrieval distractors drawn from the old topic hurt more than random ones, and whether the extract-then-answer approach of Ext2Gen removes that extra harm. The final deliverable is the generator itself, released so that results can be regenerated with fresh graphs as models change.
References
- Saving Dense Retriever from Shortcut Dependency in Conversational Search
- Beyond Continuity: Challenges of Context Switching in Multi-Turn Dialogue with LLMs
- CORAL: Benchmarking Multi-turn Conversational Retrieval-Augmented Generation
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- INVITE: a Testbed of Automatically Generated Invalid Questions to Evaluate Large Language Models for Hallucinations
- RECOR: Reasoning-focused Multi-turn Conversational Retrieval Benchmark
- TopiOCQA: Open-domain Conversational Question Answering with Topic Switching
- Aligning Extraction and Generation for Robust Retrieval-Augmented Generation
- PRCA: Fitting Black-Box Large Language Models for Retrieval Question Answering via Pluggable Reward-Driven Contextual Adapter
- Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?
- Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering