Large language models are now routinely scored on multi-turn dialogue benchmarks such as MuTual, DailyDialog, MultiWOZ and the Ubuntu corpus. Nobody has checked whether those scores reflect real conversational reasoning or recall of dialogues the model saw during training. Existing contamination detectors look for traces of memorization in individual tokens. Examples include n-gram overlap, low-probability-token scores, and loss comparisons against a reference model. Two recent results show these traces are fragile. Rewriting a test item in new words erases them, and a short round of reinforcement learning flattens them toward chance. A conversation, however, has structure above the word level: which turn follows which, who answers whom, how a request turns into a confirmation. We propose that a model which memorized a benchmark dialogue also memorized this turn-level trajectory. We further propose that this second trace survives rephrasing of every individual utterance and may survive reinforcement learning better than token likelihoods do.
The proposal borrows its central design from hierarchical recurrent encoding, the dialogue architecture in which a fast network summarizes each utterance into one vector and a slow network steps once per turn over those summaries. We use that two-timescale split as a detector rather than a dialogue model. A small hierarchical recurrent network reads, turn by turn, features extracted from the audited model's behavior under paraphrase-and-reorder counterfactuals. It then decides whether the conversation was seen in training. The contribution is a contamination test that cleanly separates what a model remembers about a dialogue's wording from what it remembers about its turn sequence. We validate it under controlled injection and under the reinforcement-learning "cloaking" regime where current detectors fail.
Core Claim
Membership signal in multi-turn benchmarks has a structural component, and this component can be isolated. Take a dialogue and rewrite every utterance with a paraphraser. Token-level memorization is then mostly destroyed, as the rephrasing results suggest. A contaminated model should nevertheless still prefer the published turn order over plausible reorderings, and should still find the original next turn unsurprising given the paraphrased history, more than a matched uncontaminated model does. Write $\ell_t(d)$ for the audited model's mean token loss on turn $t$ of paraphrased dialogue $d$, given the preceding paraphrased turns. Write $\tilde\ell_t(d)$ for the same quantity under a reference model never exposed to the benchmark. The per-turn calibrated surprise $z_t = \ell_t - \tilde\ell_t$, together with analogous gaps for locally permuted turn orders, forms a short sequence with one vector per turn. The detector is a dialogue-level recurrent network over that sequence. Its value is that it can pick up patterns such as "surprise drops sharply after the turn where a slot is first stated." A dialogue-wide average loses such patterns.
What would be new is threefold. The test targets multi-turn benchmarks, which the contamination literature has not treated. It decomposes the signal into lexical and structural parts using counterfactuals. And it tests directly whether the structural part resists the variance compression that reinforcement learning applies to flat likelihoods.
Grounding in Prior Work
The motivating evidence comes from two failure results. Test of Time turned arXiv papers into rephrased multi-step questions. It found that this removes post-cutoff decay and drops influence-function attribution from a 77.5% to a 17.5% hit rate, and it calls for "non-scalar probes." It studies only single-turn science questions and code. On The Fragility of Benchmark Contamination Detection in Reasoning Models shows that one epoch of GRPO on clean data cuts the AUROC of LiRA, a reference-model likelihood-ratio test, by about 14% from 89.1%. It also shows that all ten detectors fall to 50–58% AUROC on a contaminated distilled reasoning model. Its analysis attributes the collapse to clipping compressing the loss variance of unseen items. Both papers act as constraints on our design: a detector built on raw sequence loss is ruled out. Our structural statistic is a difference of losses across orderings of the same content, and whether it escapes that compression is the hypothesis we test, not an assumption.
The survey by Xu et al. supplies the taxonomy we use to build contamination conditions, ranging from exposure to prompts alone up to exact question-answer pairs. It also lists "sequential analysis," meaning whether a model expects items in their published order, as a comparison-based detector. That idea is the closest ancestor of ours. It operates across benchmark items, while we operate across turns within an item and remove lexical cues first. Data Contamination Quiz recasts detection as recognizing the original among word-level synonym variants, which estimated about 95% contamination of GPT-4 on XSum. It is a flat, single-instance method and serves as our black-box baseline when extended to turns. GSM1k supplies the clean-mirror logic: drops of up to 8% for some open model families, correlated with likelihood on the original at Spearman $r^2 = 0.36$. We reuse it to check whether our scores predict real performance inflation.
The method side comes from hierarchical recurrent dialogue modeling. The HRED and dual-encoder models on the Ubuntu Dialogue Corpus and DailyDialog show that a per-turn bottleneck plus a turn-level recurrence captures cross-turn context that flat recurrent language models miss. We reuse that structure as a membership classifier over per-turn features, not over words. Hierarchical Multiscale Recurrent Neural Networks and the global/local memory split in TNT give the general case for separating timescales. TNT itself studies memorization during inference, not training contamination, so it informs our intuition only. The targets of the audit come from benchmarks whose provenance makes leakage plausible. MuTual is built from publicly posted listening-comprehension exams; RoBERTa reached 71.3% against a 94.1% human ceiling. MultiWOZ-2.0, the benchmark for the HyST tracker, has never been audited for contamination even though language models are now used as zero-shot state trackers.
Research Program
The first component is a controlled contamination testbed. We take Qwen2.5-7B-Instruct, the Stage I model in the Fragility study, and split each of MuTual, DailyDialog and MultiWOZ-2.0 into member and non-member halves matched on length and turn count. Members are injected by supervised fine-tuning under three conditions. The first is verbatim dialogues. The second is dialogues with every utterance paraphrased before injection, which simulates laundering at training time. The third is the turn sequence only, with dialogue-act labels and slot transitions but heavily rewritten wording; this probes whether structure alone is learnable. Each contaminated checkpoint is then given one epoch of GRPO on clean data, reproducing the cloaking setup. A second family uses DeepSeek-R1-Distill-Llama-8B to match the Stage II regime. The reference model for calibration is the same base model fine-tuned on the non-member half of an unrelated dialogue corpus with equal token budget.
The second component is the counterfactual probe. For each test dialogue, an independent paraphraser produces three rewrites of every utterance, and we check that referents and dialogue acts are preserved using the DailyDialog act labels and the MultiWOZ slot annotations. We compute per-turn calibrated surprise under the original order and under local reorderings that keep the dialogue coherent: swapping adjacent user turns where the act labels permit, and moving a slot mention one turn later. The resulting per-turn feature vectors contain lexical-gap, order-gap and next-turn-rank features.
The third component is the hierarchical detector. An utterance-level encoder maps each turn's features, together with a frozen embedding of the paraphrased turn, to a vector. A dialogue-level GRU reads those vectors and outputs membership. To keep the detector from memorizing benchmark content itself, it is trained on controlled contamination of one benchmark and evaluated on the other two, and paraphrasers differ between training and test.
The fourth component is an audit in the wild. Public open models are scored on MuTual, and each model's detector scores are correlated with its accuracy drop on a paraphrased and turn-structure-preserving mirror of MuTual, following the GSM1k logic.
Baselines
The naive reference point is chance at 50% AUROC. Alongside it we run the dialogue-level flat loss: total loss of the verbatim dialogue, thresholded. This is what a practitioner would try first, and any method must clearly beat it on the paraphrased conditions to matter.
The strongest existing method is LiRA run on full dialogues against the same reference model we use. It reports 89.1% AUROC before reinforcement learning, falling by about 14% after one epoch of GRPO, and the ten-detector suite averages 50–58% AUROC on the Stage II model (Fragility). No reported result was found on Lacuna for any contamination detector on a multi-turn dialogue benchmark, so these single-turn numbers are a reference, not a target. We will rerun LiRA, Min-K%, and Loss on our checkpoints with identical member and non-member splits, the same reference model, and equal hyperparameter search budgets. Beating LiRA means an AUROC higher than LiRA's by a margin outside bootstrap confidence intervals on the paraphrased-injection and post-GRPO conditions, while staying within a few points of it on verbatim pre-RL injection.
We also run DCQ extended to dialogues. The model picks the original next turn among synonym-perturbed variants, with its bias-compensation step applied. This is the strongest black-box competitor.
Two ablations isolate the key ingredients. The first keeps the same per-turn features but replaces the dialogue-level recurrence with mean pooling into logistic regression. The second keeps the recurrence but drops the order-gap features, leaving only paraphrased lexical gaps. If the full detector does not beat both, neither the hierarchy nor the structural signal earns its place.
Why not just…?
Why not just run published-order sequential analysis on whole dialogues? It is the closest existing idea in the survey, and a single order-gap scalar averaged over turns may capture most of our signal. This is a real threat, and it is exactly the mean-pooling ablation. If pooled order gaps match the recurrent detector, we report that the contribution is the paraphrase-then-reorder statistic, not the hierarchical encoder, and drop the encoder.
Why not just build a clean paraphrased mirror of each dialogue benchmark, like GSM1k? A mirror measures inflated accuracy for a whole model. It does not say which items were seen, and it requires costly matched construction per benchmark. Test of Time also shows that rephrasing can preserve memorized reasoning, so a small mirror drop does not imply cleanliness. We build a MuTual mirror anyway as the external check on our scores rather than as a competitor.
Why not just use LiRA or Min-K% on the full dialogue? On verbatim, pre-RL contamination they may well suffice, and we expect parity there. The evidence against them comes from the regime that matters, rephrased items and post-RL models, where Fragility reports collapse to near chance. Our post-GRPO conditions measure directly whether order-gap differences resist the same compression. If they collapse equally, the core claim is false.
Why not just extend DCQ to turns? DCQ detects recognition of exact wording. Once utterances are paraphrased, its original-versus-synonym choice has no original left to recognize. The paraphrased-injection condition tests this directly.
Decisive Checks
The idea stands or falls on three outcomes. First, under paraphrased injection, the structural features must separate members from non-members above chance while lexical features alone do not. Second, after one GRPO epoch, the structural detector's AUROC must degrade by clearly less than LiRA's measured drop on the same checkpoints. Third, trained on one benchmark, the detector must transfer to the other two, and in the wild its scores must correlate positively with the MuTual mirror accuracy drop. A useful null result is also possible. If the turn-sequence-only injection produces no detectable structural trace and no accuracy gain, then multi-turn benchmarks are mainly vulnerable at the wording level, and existing flat detectors plus utterance paraphrasing are the right tools.
References
- Test of Time: Rethinking Temporal Signal of Benchmark Contamination
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- Benchmark Data Contamination of Large Language Models: A Survey
- Data Contamination Quiz: A Tool to Detect and Estimate Contamination in Large Language Models
- A Careful Examination of Large Language Model Performance on Grade School Arithmetic
- Training End-to-End Dialogue Systems with the Ubuntu Dialogue Corpus
- DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset
- Hierarchical Multiscale Recurrent Neural Networks
- TNT: Improving Chunkwise Training for Test-Time Memorization
- MuTual: A Dataset for Multi-Turn Dialogue Reasoning
- HyST: A Hybrid Approach for Flexible and Accurate Dialogue State Tracking