Some newer language models do not write text left to right. Masked diffusion models start from a fully blanked-out answer and fill in the blanks over many steps, usually filling first whichever position they are most confident about. Any-order autoregressive models such as A3 can also generate positions in a confidence-chosen order. That choice of order is itself a readout of what the model already "knows." We conjecture that when a benchmark item has leaked into training, the model fills in the final answer, such as the number at the end of a math problem or the key finding in a radiology impression, early in decoding, before the reasoning or context that should support it. On an item the model has genuinely never seen, the answer should come late. Current contamination tests score whole datasets. The answer's position in the decoding order could instead flag individual items and even individual answer spans.
A second part of the proposal uses in-context examples as a probe. Adding worked examples to the prompt pushes a model out of a verbatim-memorized groove. It also creates a failure of its own: with many examples, models stop reading the question and copy label statistics from the prompt. Both effects can make an answer appear early, so a usable detector has to tell them apart. We propose a three-way diagnostic, validated with planted contamination on math, science, and radiology-summary tasks. It separates "memorized," "copying from the prompt," and "actually derived," and it scores clinical outputs with graph-based clinical metrics rather than n-gram overlap. We also test whether easy-first decoding makes contaminated benchmarks look better than they should.
Core Claim
Consider a model that decodes in confidence order. We define the answer onset of an item as how early its answer tokens are committed, relative to the rest of the output. Let $s_i$ be the decoding step at which position $i$ is unmasked, $S$ the total number of steps, and $A$ the set of answer positions. Then
$$\rho = \frac{1}{|A|\,S}\sum_{i \in A} s_i ,$$
so small $\rho$ means the answer was written before its supporting text.
We claim three things. First, on its own $\rho$ is confounded by answers that are simply formulaic, such as "No acute cardiopulmonary process." Two contrasts remove the confound: $\rho$ on the original item versus a meaning-preserving paraphrase, and $\rho$ at zero shots versus $k$ shots. Together they give a per-item contamination signal. Second, a third contrast, swapping in an unrelated query, separates memorization from many-shot label copying. Memorized answers are early and query-specific. Prompt-copied answers are early and query-insensitive. Derived answers are late. Third, easy-first decoding widens the accuracy gap between contaminated and clean items compared with random-order or information-gain decoding. That would make it a biased way to evaluate diffusion models on public benchmarks.
Why This Is Not Already Done
The closest work is CoDeC. It observes that in-context examples raise the model's likelihood on unseen data but lower it on memorized data, and turns this into a dataset-level score with 99.9% AUC across 40+ models. CoDeC uses left-to-right likelihood and reports at the dataset level. We reuse its core insight that context disrupts memorization, but read the effect from the unmasking trajectory, per item and per span.
PaCoST compares the model's confidence that an answer is correct on original versus rephrased questions using a paired t-test. We borrow its paraphrase contrast as our control for formulaic answers. Yax et al. show that contaminated puzzles are completed with near-zero token surprise. Their work motivates the premise that memorized text is over-confident, but it measures surprise along a fixed order.
On the decoding side, The Confidence Shortcut shows that confidence-ordered masked diffusion models commit locally predictable digits before carries arrive. On long carry chains, PAPL collapses to 2.6% while random masking holds 99.2%. Info-Gain Sampler replaces greedy confidence with information gain and gains 2.9 to 11.6 points on GSM8K and MATH-500. Both papers treat early commitment as a reasoning error on clean data. We treat it as evidence about the training data, and we ask whether contamination feeds exactly the shortcut they describe. A3 supplies an any-order model whose order behavior we can compare against native diffusion models.
On saturation, the ICL-paradox work of Elazar et al. finds that attentiveness on MNLI drops from 92.9% to 62.0% between 0 and 3 shots, measured with a counterfactual query-swap test. We reuse that test directly as our third contrast. Upasani et al. show saturation points vary from about 4 shots for chain-of-thought on GPQA Diamond to 50–70 shots per class on Banking77. This constrains us to sweep shot counts rather than fix one.
Research Program
We use open confidence-ordered decoders: Dream and DiffuLlama as native masked diffusion models, and A3-8B as an any-order autoregressive model. We run each under three orderings: greedy confidence, Info-Gain, and random. For every item we log the full unmasking trajectory. When a model-level likelihood is needed, we compute exact likelihoods with DUEL.
Ground truth for contamination comes from planting it. We continue training each model on its native objective with a known random half of GSM8K test, MATH-500, and GPQA Diamond. Exposure is varied (1, 4, 16 repetitions) along with leak type: verbatim, paraphrased, and answer-only. The other half stays clean. WMDP, used by PaCoST as a recent benchmark, serves as an additional presumed-clean set. We will check release dates against each model's data cutoff before relying on it.
For the clinical regime we fine-tune Dream on a text-only MIMIC-CXR task that turns the Findings section into an Impression, using ViLMedic for data handling. We plant a subset of test impressions the same way. Answer spans are defined by RadGraph observation entities and their present, uncertain, or absent status. The RadGraph parser reaches 0.94 entity F1 and 0.82 relation F1, which makes it accurate enough to define spans. Onset is then computed per clinical finding rather than per report. Outputs are scored with RadGraph F1 and RadCliQ, which correlates with radiologist error counts at Kendall-tau b = 0.522. This lets us ask a question BLEU cannot answer: when a flagged finding is removed or paraphrased, does clinical fidelity drop, or only surface overlap?
The detector combines the three contrasts: paraphrase shift in $\rho$, demonstration shift in $\rho$ across a shot sweep of 0, 1, 4, 16, and 64 (plus 50–70 per class on Banking77 as a saturation control), and query-swap sensitivity of the early answer. We fit it as a small logistic model on one benchmark and test it on the others, so it is never tuned on the evaluation benchmark.
Baselines
The naive reference point is n-gram overlap between test items and the planted training data. It should catch verbatim leaks and miss paraphrased and answer-only leaks. It also sets the floor for per-item AUC.
The strongest existing detector is CoDeC, which reports a dataset-level AUC of 99.9%. We found no reported per-item AUC on Lacuna for CoDeC. We will therefore run it two ways on the same models, items, and shot sweep. One version uses DUEL exact likelihood in place of left-to-right likelihood. The other version runs on a size-matched left-to-right model trained with the same planted contamination, so it gets equal exposure. Beating it means higher per-item AUC at matched false-positive rate, especially on paraphrased and answer-only leaks. At the dataset level we expect only to match it.
PaCoST is run with the same paraphrases we use. It reports significance flags (p < 0.05) rather than AUC. We found no reported AUC on Lacuna, so we compute one from its per-item confidence differences.
Token-level surprise in the style of Yax et al. is computed from DUEL likelihood on answer tokens. The key ablation is a "single-point answer probe": the model's confidence in the answer given only the question, with the derivation fully masked. This isolates whether the trajectory adds anything beyond one well-chosen conditional probability. A second ablation drops the query-swap contrast, to show how often many-shot label copying is then misread as contamination. For the decoding-bias claim, the comparison is contaminated-minus-clean accuracy under greedy confidence, Info-Gain, and random order, holding the number of decoding steps equal.
Why not just…?
Why not just run CoDeC on the diffusion model's likelihood? CoDeC's signal is an aggregate proportion of items whose likelihood drops. It was designed for, and reported at, the dataset level. It cannot say which finding in a radiology impression was memorized. It also does not separate a context-induced drop caused by memorization disruption from one caused by the model attending to prompt labels, the effect Elazar et al. document. The CoDeC-on-DUEL baseline tests this directly. If it matches our per-item and per-span AUC, the order signal adds nothing and we will say so.
Why not just measure the model's confidence in the answer with the reasoning masked out? This is the most serious threat, because $\rho$ may be largely a function of that one number. The single-point probe ablation is the decisive experiment. We expect the trajectory to add information in two places. The first is long, multi-finding outputs such as impressions, where "which spans come first" matters. The second is the shot sweep, where onset shifts track memorization disruption. If the probe performs equally well, the contribution narrows to that cheaper detector plus the decoding-bias finding, and we will report it that way.
Why not just audit a left-to-right model and ignore any-order decoders? Left-to-right auditing tools already exist. But the object of study here is whether easy-first decoding inflates benchmark scores for diffusion models, which a left-to-right audit cannot address. The Confidence Shortcut already shows that order changes outcomes dramatically on clean data. The contaminated-versus-clean accuracy gap across the three orderings answers the question directly.
Decisive Checks
The central test uses the planted-contamination benchmarks. The three-contrast detector must beat CoDeC-on-DUEL and the single-point probe in per-item AUC on held-out benchmarks, with paraphrased and answer-only leaks reported separately.
On MIMIC-CXR there are two checks. Per-finding flags should localize planted entities, measured with RadGraph entity-level precision against the planted set. And rephrasing the flagged findings should lower RadGraph F1 and RadCliQ more than it lowers BLEU. That pattern would confirm that the memorized content is clinical, not stylistic.
On the saturation control, Banking77 at 50–70 shots per class with no planted contamination, the query-swap contrast must keep false positives near the clean-data rate. Without it, false positives should rise with shot count.
For the decoding-bias claim, the contaminated-minus-clean accuracy gap must be larger under greedy confidence than under random and Info-Gain ordering across exposure levels. A null result here would itself be informative: it would mean easy-first evaluation is not disproportionately exploiting leaks.
References
- Detecting Data Contamination in LLMs via In-Context Learning
- PaCoST: Paired Confidence Significance Testing for Benchmark Contamination Detection in Large Language Models
- Studying and improving reasoning in humans and machines
- The Confidence Shortcut: A Reasoning Failure Mode of Masked Diffusion Models
- Improving Sampling for Masked Diffusion Models via Information Gain
- Autoregressive Models Rival Diffusion Models at ANY-ORDER Generation
- Measuring and Improving Attentiveness to Partial Inputs with Counterfactuals
- Test-Time Adaptation via Many-Shot Prompting: Benefits, Limits, and Pitfalls
- DUEL: Exact Likelihood for Masked Diffusion via Deterministic Unmasking
- RadGraph: Extracting Clinical Entities and Relations from Radiology Reports
- Evaluating progress in automatic chest X-ray radiology report generation