When a language model has seen a benchmark's answer key during training, it often reproduces the specific choices that key made rather than just producing a correct answer. A summary benchmark's gold reference picks particular facts in a particular order. A math benchmark's worked solution takes one route through the problem among many valid ones. A clean model has no reason to prefer the gold version over an equally good, independently written alternative. A contaminated model does. This note proposes measuring that preference directly. For each test item we collect several equally valid alternative references and ask whether the model's output sits measurably closer to the gold reference's idiosyncratic content than to the alternatives'.
What makes this a cross-lingual problem is that the leak may have happened in another language. Data Contamination Can Cross Language Barriers showed that fine-tuning on French, Spanish or Chinese translations of English benchmarks inflates English scores. In one case, LLaMA3-8B on MathQA rose from 42.01% to over 95% after French contamination, while n-gram and likelihood checks saw nothing. The memorized key then resurfaces as English output that shares meaning but not words with the English gold. Detecting it therefore requires a comparator that matches content across languages and phrasings while ignoring style. Building such comparators is exactly what the cross-lingual text comparison literature has worked on.
Core Claim
The claim is that per-item preference for the gold reference over matched alternatives is a contamination signal with four useful properties. It works on open-ended generation, which multiple-choice tests such as Choice Confusion cannot handle. It needs only the model's text output, with no log-probabilities. It comes with its own null distribution, built from exchangeability among valid references, so no panel of reference models is needed. It yields an item-level estimate of how much accuracy is attributable to leakage.
We further hypothesize that the signal survives the reinforcement-learning post-training that erases likelihood-based detectors. The reason is that it reads behavior, not loss. This is the most important thing to test.
Let $y$ be the model's output for an item, $r_0$ the gold reference, and $r_1,\dots,r_k$ independently produced valid references. Let $s(y,r)$ be a content-similarity score between output and reference. The per-item statistic is the rank of the gold reference:
$$R = \#\{j \in \{1,\dots,k\} : s(y,r_0) > s(y,r_j)\}.$$
If the $k+1$ references are exchangeable from the model's point of view, $R$ is uniform on $\{0,\dots,k\}$. Exchangeability here means the gold is just one valid answer among several and the model never saw it. Averaging $R$ over a benchmark's items gives a test with an exact permutation null. The excess mass at $R=k$ estimates the fraction of items on which the model reproduces the key.
How this differs from existing work
Five pieces of prior work set the problem up, and the proposal differs from each.
Data Contamination Can Cross Language Barriers motivates the problem. It also supplies the contamination protocol we reuse: fine-tuning on translated test sets in seven languages. Its detector, Choice Confusion, swaps multiple-choice distractors for other items' correct answers, so by construction it cannot touch generation. The authors name open-ended formats as future work. Our test carries Choice Confusion's logic, a clean model being indifferent to substitutions that a contaminated model is not, from answer options over to whole references.
Contamination Report for Multilingual Benchmarks audits multilingual models with a canonical-order exchangeability test. It finds Llama-3.1-8B and Gemma-2-9B contaminated on nearly every benchmark except, for Llama 3.1, XLSum. The test is binary, costs about 362 hours on 8×H100, and only detects leaks that preserved file order. Our exchangeability argument is over references rather than item orderings, so shuffled or individually scraped leaks remain visible.
ConStat quantifies inflation by comparing scores on original and rephrased benchmarks against a difficulty curve fitted over trusted reference models. It is the strongest quantitative competitor, but it needs that panel and operates at the benchmark level.
Soft Contamination Means Benchmarks Test Shallow Generalization finds semantic duplicates in English training data using embedding cosine plus an LLM judge. It works on the corpus side and only in English.
On The Fragility of Benchmark Contamination Detection in Reasoning Models constrains the design. It shows that one epoch of GRPO cuts LiRA and loss-based detection AUROC by roughly 14–16 points, and that ten detectors sit at 50–58% AUROC on contaminated reasoning models. Any likelihood-based design is therefore ruled out from the start.
The cross-lingual comparison literature supplies the method. The SemEval-2022 Task 8 systems learned to score news-article similarity across ten languages while separating content dimensions (entities, time, narrative) from style and tone. HFL at SemEval-2022 Task 8 did this with translate-train augmentation, multi-dimensional regression and an MSE variant of R-Drop consistency regularization, scoring 0.818 Pearson. WueDevils at SemEval-2022 Task 8 instead aligned articles through all-pairs sentence similarity matrices. MUSTS constrains our encoder choice. It reports that LaBSE and even SIF beat LLM prompting and LLM embeddings on low-resource languages like Sinhala and Tamil, and that MTEB rankings do not carry over. We will not assume an LLM judge or a leaderboard-topping embedder is the right comparator.
Research Program
The core artifact is the comparator $s$. A plain cosine between whole-text embeddings is the wrong tool, because two valid summaries of the same article are topically near-identical. MUSTS flags exactly this confusion of relatedness with equivalence. What separates a memorized key from an alternative is the content the key chose and the alternatives did not.
We therefore define $s(y,r_0)$ as idiosyncratic content recall. We split each reference into content units, meaning sentences or solution steps with their intermediate quantities and entities. We mark the units of $r_0$ that no alternative reference matches, then compute the fraction of those gold-specific units that are matched by some unit of $y$. Matching uses a cross-lingual sentence similarity matrix in the style of WueDevils, built on a multilingual encoder. That encoder is fine-tuned with HFL-style translate-train augmentation and consistency regularization, so that matching is invariant to language and phrasing. Numbers and named entities are matched exactly after normalization, because they are the anchors that survive translation. The same construction is applied symmetrically to each $r_j$, so the rank statistic compares like with like.
Alternative references are where the null can fail, so they get the most care. For XLSum we commission independent summaries of the source article. For GSM8K and MathQA we collect independent solution write-ups that take different but valid orderings and decompositions. In both cases human writers produce one subset, and several different LLMs produce a larger subset under diverse prompts. The human subset is the anchor for checking the larger LLM-produced set.
Exchangeability can break in two directions. Gold references may be more typical of the source than alternatives, which would produce false positives. Alternatives written by LLMs may be closer to any LLM's output style, which would produce false negatives. We calibrate against both. Models released before a benchmark existed serve as negative controls. Items written after model training cutoffs serve as clean references. The human-only subset provides a check that conclusions do not depend on who wrote the alternatives.
The program has three stages. First, a controlled study. We fine-tune open 7–8B models on translated gold references for XLSum, GSM8K and MathQA in the seven languages of the cross-language-barrier paper, at contamination rates from full test sets down to the ecologically realistic levels used in the soft-contamination study (about 4 in 10,000 training points). We then evaluate in English. We also run the untested reverse direction: contaminate in English and evaluate on the native-language XLSum splits.
Second, a cloaking study. We apply one epoch of GRPO on clean data after contamination, as in the fragility paper, and ask whether the rank statistic survives while likelihood detectors collapse.
Third, an audit of released models. We run the test on Llama-3.1-8B, Gemma-2-9B and Aya-23-8B and compare against the exchangeability verdicts already published for them.
A secondary analysis asks whether detectability tracks language distance. We stratify contamination languages by the DistaLs genealogical, Grambank, lexical and resource-availability measures. This bears on an unresolved tension between two findings. The cross-language-barrier paper found stronger transfer from European languages. The language contamination study found transfer driven by data volume more than by similarity to English.
Baselines
The naive reference point is n-gram overlap between the output and the gold reference, tested the same way against alternatives. This is the rank test with ROUGE-style overlap as $s$. It is the obvious thing a practitioner would try, and it should work for same-language contamination and fail for cross-lingual contamination. We expect it to stay near chance on translated leaks, consistent with the cross-language-barrier paper's finding that surface checks miss them. Beating it there is the minimum bar.
The strongest existing quantitative method is ConStat. It reports a 98.4% detection rate on sample-specific contamination in controlled trials, an inflation of δ≈8.3% for Mistral-7B on GSM8K, and δ≈6.6% for Llama-3-70B on ARC (ConStat). We run it on identical contaminated checkpoints and benchmarks, with rephrased reference sets built from the same items. Its reference-model panel uses the same open models available to us. We report its detection rate and inflation estimate beside ours.
Beating ConStat means at least one of three outcomes, each required to hold under matched compute. First, higher detection AUROC on cross-lingually contaminated checkpoints at low contamination rates. Second, inflation estimates closer to the known ground truth, which the controlled study gives us because we know which items were leaked. Third, retained power after GRPO where ConStat's does not.
For multiple-choice-compatible data, MathQA, Choice Confusion is the direct competitor. The reported numbers on MMLU are a +26.25-point gain for clean LLaMA3 models and a drop of about 17 points for contaminated ones (Data Contamination Can Cross Language Barriers). We run both methods on the same MathQA checkpoints and expect parity there. Our gain is that the rank test also covers GSM8K and XLSum, where Choice Confusion does not apply.
The likelihood detectors Min-k% Prob, Loss and LiRA are run on the post-GRPO checkpoints with the fragility paper's setup. Their reported post-RL AUROC is 50–58%, and that is the floor we expect them to hit again.
For the generative cross-lingual setting itself, no reported result was found on Lacuna for detecting translated contamination on XLSum or GSM8K. The controlled study will produce the first such numbers.
The key ablation replaces idiosyncratic content recall with whole-text cosine from an off-the-shelf multilingual encoder, LaBSE, keeping everything else fixed. LaBSE is chosen because MUSTS found it the strongest unsupervised option on low-resource languages. A second ablation drops the alternative references and thresholds raw similarity to the gold. This isolates how much the exchangeability null contributes over a tuned cutoff.
Why not just…?
Why not just translate everything into English and use n-gram matching? This answers a different question, about corpus filtering rather than auditing a trained model. It also fails on the model side: a memorized French key regenerated in English shares meaning with the English gold, not its exact strings. Our naive baseline is essentially this test with surface overlap as the score. The controlled study will show directly whether it detects translated leaks. If it does, the comparator adds nothing, and we would report that.
Why not just use a multilingual embedding cosine, as the soft-contamination audit did? This is the real threat. The ablation is designed to settle it. Whole-text cosine should register two valid summaries of the same article as near-identical, which buries the signal in topical relatedness. That is the conflation MUSTS warns about. If LaBSE cosine nonetheless matches idiosyncratic recall in the rank test, the contribution reduces to the test design. We would then recommend the simpler comparator.
Why not just run ConStat with rephrased benchmarks? ConStat is the closest competitor and may perform comparably on in-language contamination. Our expected advantages are specific. We need no reference-model panel, whose own contamination is a confound. We get an item-level rather than benchmark-level estimate. And a rephrased question still elicits the memorized answer, whereas a swapped reference does not. The head-to-head on identical checkpoints decides this, including after GRPO.
Why not just ask a strong LLM whether the output looks memorized? The judge may itself have seen the benchmark, which is the same self-reference problem raised in the soft-contamination and ConStat work. MUSTS also shows that LLM similarity judgments degrade on low-resource languages, which is exactly where cross-lingual leaks would be hardest to spot.
Proposed Experiments
The decisive check is the cloaking study. We take checkpoints contaminated through translated keys, confirm their inflated scores persist after one epoch of clean GRPO, and measure detection AUROC for the rank test against Min-k%, LiRA and ConStat on the same checkpoints. If the rank statistic falls to chance along with the likelihood detectors, the behavioral-signal hypothesis is wrong. In that case the method reduces to a generation-capable but fragile alternative to Choice Confusion.
The second check is calibration. On negative-control models and post-cutoff items, the distribution of $R$ must be indistinguishable from uniform, both with human-written and with LLM-written alternatives. Otherwise every positive finding is suspect.
The third check is quantification. In the controlled study, the estimated fraction of reproduced items should track the true leaked fraction across contamination rates. Accuracy on items flagged as reproduced versus unflagged should recover the known inflation.
The audit of Llama-3.1-8B, Gemma-2-9B and Aya-23-8B on XLSum provides an external sanity check, since their exchangeability verdicts are already published. Disagreement there is informative rather than fatal, because the two tests detect different leakage routes. Finally, the DistaLs stratification reports whether detectability and inflation follow linguistic distance or the volume of leaked data. This tests the two competing explanations in the prior work on a reasoning-level task rather than on part-of-speech tagging.
References
- Data Contamination Can Cross Language Barriers
- Contamination Report for Multilingual Benchmarks
- ConStat: Performance-Based Contamination Detection in Large Language Models
- Soft Contamination Means Benchmarks Test Shallow Generalization
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- HFL at SemEval-2022 Task 8: A Linguistics-inspired Regression Model with Data Augmentation for Multilingual News Similarity
- WueDevils at SemEval-2022 Task 8: Multilingual News Article Similarity via Pair-Wise Sentence Similarity Matrices
- MUSTS: MUltilingual Semantic Textual Similarity Benchmark
- DistaLs: a Comprehensive Collection of Language Distance Measures
- Language Contamination Helps Explains the Cross-lingual Capabilities of English Pretrained Models