A language model can score well on an English test because it saw that test during training, even when what it saw was a French or Chinese translation. Data Contamination Can Cross Language Barriers showed this directly. Fine-tuning LLaMA3-8B on a French translation of MathQA lifted its English MathQA accuracy from 42.01% to above 95%. Standard leak detectors, which look for overlapping word sequences or unusually low perplexity, missed it entirely, because the leaked text shares no surface strings with the English test. Current tools can sometimes say that a model is contaminated. None of them can say how many points of its score come from the leak, or through which language the leak arrived. Both questions matter for multilingual evaluation, where developers scrape uncurated non-English web text and benchmark files leak heavily (Contamination Report for Multilingual Benchmarks).
We propose a black-box probe built from a multilingual semantic network, meaning a graph such as BabelNet that links word senses across languages. Languages carve up meaning differently. English "river" and "stream" may map to one word in one language and to two in another. Suppose a model's memory of a test item is keyed to how that item was worded in a source language L. Then rewording the English item in a way that L cannot distinguish should leave the memory intact, and rewording it in a way L does distinguish should break it. The probe compares the model's accuracy drop under these two kinds of edits, language by language. That comparison yields a per-language attribution score and an estimate of how much accuracy the leak inflated. It works for any task where the answer can be checked, not only multiple choice.
Core Claim
Cross-lingual contamination leaves a fingerprint in the lexical geometry of the source language. We make three edits to each benchmark item, all chosen to leave the correct answer unchanged.
The first is a within-concept swap: a synonym from the same BabelNet synset (a set of words that all express one sense). The second is a sibling-concept swap: a different concept under the same parent category, used only on words the answer does not depend on, such as "pears" for "apples" in an arithmetic problem. The third splits the within-concept swaps for each candidate language L into two groups, using BabelNet's sense lexicalizations in L. In the L-collapsed group, the original and substitute words translate to the same word in L. In the L-distinguished group, they translate to different words.
The three hypotheses about what a model has learned make different predictions.
- Genuine ability. Performance does not change under any edit.
- Memorization of the English string. Performance drops under every edit, roughly equally across collapsed and distinguished swaps.
- Memory keyed to source language L. Performance survives L-collapsed swaps but drops under L-distinguished swaps and under sibling swaps.
We define an attribution score for each candidate language $L$:
$$A_L = \Delta^{\text{dist}}_L - \Delta^{\text{coll}}_L,$$
where $\Delta^{\text{dist}}_L$ and $\Delta^{\text{coll}}_L$ are the model's accuracy drops, relative to the unedited items, on swaps labelled L-distinguished and L-collapsed respectively. The design has one important property. The same swap set is relabelled for each language. A swap that is collapsed in French may be distinguished in Japanese. So any property of the swaps themselves, such as how close the synonyms are or how frequent they are, cancels when $A_L$ is compared across languages. A clean model and a model that memorized the English text should both give $A_L$ near the clean-model null for every L. A model whose leak came through L should give a peak at L.
We separately estimate inflation as
$$I = (\text{acc}_{\text{orig}} - \text{acc}_{\text{sib}}) - (\text{acc}^{\text{clean}}_{\text{orig}} - \text{acc}^{\text{clean}}_{\text{sib}}).$$
Here $\text{acc}_{\text{orig}}$ and $\text{acc}_{\text{sib}}$ are accuracy on original and sibling-edited items. The superscript "clean" marks the same quantities for an uncontaminated reference model. That second term subtracts whatever genuine difficulty the edit itself introduces.
The probe also answers a scientific question the literature leaves in tension. Data Contamination Can Cross Language Barriers treats language as an interchangeable interface over language-agnostic knowledge. It attributes the stronger leakage from European languages to shared subwords and "closer conceptual mappings," without separating the two. Two other papers suggest representations are not symmetric across languages. Cross-Lingual Consistency finds that reasoning gets trapped in language-specific errors that sampling in other languages can escape. Language models are multilingual chain-of-thought reasoners finds that reasoning in English helps even on non-English inputs. If memorized items really are stored in a language-agnostic way, $A_L$ will be flat across languages even under known French contamination. That would be a clean negative result supporting Yao et al.'s view. A sharp peak would show that memory carries the lexical boundaries of its source language.
Research Program
Edit generator. The first artifact is a generator that takes a benchmark item and produces the three edit families. It works as follows.
- It tags content words with BabelNet senses through the BabelNet API (Multilingual WSD with Just a Few Lines of Code), using the multilingual joint sense-disambiguation scoring of Joining Forces Pays Off to choose senses.
- It proposes synonym substitutes from the same synset and sibling substitutes from the WordNet taxonomy.
- It labels each synonym swap as collapsed or distinguished for each of eight candidate source languages: English plus the seven languages in Yao et al. (Chinese, French, German, Italian, Japanese, Korean, Spanish). A swap is labelled collapsed in L when both words share a lexicalization in L. This uses the same cross-lingual intersection idea by which BabelRelate! prunes spurious senses.
- It checks each label by translating the original and edited items into L with two independent machine-translation systems. A swap counts as collapsed only if both translations are identical up to the swapped word.
BabelNet's coverage of verbs and adjectives is weaker than its coverage of nouns (Joining Forces Pays Off), so we restrict swaps mainly to nouns. We will report the number of usable swaps per item and per language, because that number limits statistical power. Annotators will check a random sample of edits to confirm the answer is preserved.
Controlled ground truth. The second component is a set of models with known contamination, built following Yao et al.'s protocol. We fine-tune LLaMA3-8B and Qwen1.5-7B on translated MathQA, MMLU and ARC-Challenge in each of the seven languages and directly on English. We also build a "blended" condition that mixes fragments from two languages, which Yao et al. leave open. Matched clean controls are fine-tuned on an equal volume of unrelated text in the same language, so that language exposure alone is not mistaken for leakage. True inflation is then known: it is contaminated accuracy minus clean-control accuracy. True source language is known by construction.
Mechanism analysis. The third component asks why some languages leak more strongly. We fit item-level transfer, measured as the English accuracy gain on each item, against two predictors: subword overlap between the English item and its L translation, and the fraction of the item's content-word senses that map one-to-one into L. The second predictor is the kind of concept-overlap quantity proposed in Lexical semantic typologies from bilingual corpora. This tests the two explanations that Yao et al. leave entangled.
Baselines
Raw accuracy and n-gram overlap. The naive reference point is raw benchmark accuracy with n-gram overlap against any available pretraining text. Yao et al. report that overlap returns nothing on translated leaks. It serves as a floor, and as the "do nothing" estimate of inflation, which is zero.
Min-K%++, run per language. The strongest likelihood-based competitor is Min-K%++. It scores a text by how far each token's log-probability stands above the model's average over its vocabulary, normalized by the spread. It reports AUROC gains of 6.2% to 10.5% over Min-K% on WikiMIA, a benchmark that separates text a model has seen from text it has not. We give it the fairest possible version of the attribution task: we run it on the English items and on our own translation into each candidate language, and attribute the leak to the language with the highest membership score. No reported result on cross-lingual detection or attribution was found on Lacuna for Min-K%++.
Choice Confusion. On multiple-choice benchmarks the strongest existing detector is Choice Confusion from Data Contamination Can Cross Language Barriers. It replaces each question's wrong options with correct answers taken from other questions. The paper reports that a clean LLaMA3 gained 26.25 points on MMLU under this change, while a contaminated one dropped 17 points. We run it on exactly the same items and models. It cannot attribute a source language and does not apply to free-form MathQA answers, so on those comparisons only detection AUROC is shared.
The exchangeability test. On in-the-wild models the reference is the Black Box test from Contamination Report for Multilingual Benchmarks. It checks whether a model performs anomalously well when items appear in their canonical published order rather than shuffled, flagging contamination at $p < 0.0013$. It costs 362 hours on 8 H100 GPUs and gives a binary verdict. We do not rerun it; we use its published flags as an external label.
Ablations. Two ablations isolate the key ingredient. In the first, we keep the same number of synonym swaps but assign the collapsed/distinguished labels at random, which removes the semantic network's lexical boundaries. In the second, we label swaps by subword overlap in L instead of sense collapse.
All methods see the same items, the same models and the same query budget. Any hyperparameters, such as the K in Min-K%++ or our swap-sampling thresholds, are tuned on held-out contaminated and clean models, never on the evaluation models.
What counts as success. Our method beats the baselines if it achieves three things on held-out models: higher top-1 attribution accuracy over the eight candidate languages than per-language Min-K%++ (chance is 12.5%); detection AUROC at least matching Choice Confusion on multiple-choice items; and an inflation estimate $I$ whose error against true inflation is small relative to the 5 to 10 point MMLU gains Yao et al. report.
Why not just…?
Why not just translate the test into each candidate language and run a membership detector on each translation? This is the most serious threat, which is why it is a main baseline. Its weakness is that our translation is unlikely to match the leaked translation word for word. Min-K%++ rewards the exact token sequence that formed a local probability peak during training, so it is sensitive to surface wording. Our probe never needs to reproduce the leaked text. It needs only the language's sense boundaries, which hold whatever translation was leaked. If per-language Min-K%++ attributes as well as $A_L$ in the controlled setting, the semantic network adds nothing for attribution, and we will report that plainly.
Why not just paraphrase items with a language model and measure the accuracy drop, as evolved benchmarks do? Benchmarks evolved from HumanEval drop by 39.4% on average according to the contamination survey, and graph-generated problems in DyVal push supposedly strong models to near zero. A paraphrase drop shows that a model is fragile, but it cannot separate memorization of the English string from a memory keyed to another language, and it cannot name the source. The randomly labelled ablation is exactly this alternative with matched edit volume. If it attributes as well as the full method, our central idea fails.
Why not just compare the model's accuracy across languages and blame the language where it does best? Accuracy in each language mostly reflects general ability in that language. Language Contamination Explains the Cross-lingual Capabilities of English Pretrained Models found that ability tracks how much of each language was in pretraining, and Yao et al. show that leaked items inflate scores in other languages too. Because $A_L$ applies all edits within English and only changes which language's boundaries are used to label them, ability in each language drops out. We will still run this accuracy-profile heuristic on the controlled models to measure how often it misattributes.
Why not just rely on Choice Confusion or the exchangeability test? They answer different questions. Choice Confusion needs multiple-choice options. The exchangeability test needs the benchmark's canonical file order to have leaked, needs heavy compute, and returns only a yes/no verdict. Neither gives points of inflation or a source language. Both remain as external checks in our experiments.
Proposed Experiments and Decisive Checks
Controlled study. On MathQA (free-form numeric answers, with sibling swaps on answer-irrelevant nouns), MMLU and ARC-Challenge, we compute $A_L$ for all eight candidate languages on every contaminated and clean model. The first decisive check is whether $A_L$ peaks at the true source language well above the clean-model null. If $A_L$ is flat under known French contamination, that supports the language-agnostic view of memory and shows that the attribution route is closed. That outcome is informative, not a failure of the study. The second check compares the error of $I$ against true inflation with that of every baseline. The third check, on blended models, asks whether $A_L$ recovers both source languages with peak heights that track their mixing proportions.
Explaining the European-versus-East-Asian gap. We fit item-level transfer jointly on subword overlap and one-to-one sense mapping. If sense mapping explains variance after controlling for subword overlap, the "closer conceptual mappings" explanation gets measurable support. If it does not, the gap is a tokenizer effect.
In the wild. We apply the probe to Llama-3.1-8B, Gemma-2-9B and Aya-23-8B on XCOPA and PAWS-X, using English-pivot and native-language items. We compare our verdicts with the published exchangeability flags: the first two models are flagged on most benchmarks, while Aya-23 passes on XCOPA and PAWS-X. Where the flags agree, the new contribution is the estimate of inflated points and the most likely source language. Where they disagree, we inspect items by hand. We also run MGSM (Language models are multilingual chain-of-thought reasoners). Its 250 problems are translations of English GSM8K problems, so it directly tests a question that paper leaves open: whether high non-English scores partly come from the English source leaking through.
References
- Data Contamination Can Cross Language Barriers
- Contamination Report for Multilingual Benchmarks
- Cross-Lingual Consistency: A Novel Inference Framework for Advancing Reasoning in Large Language Models
- Language models are multilingual chain-of-thought reasoners
- Multilingual WSD with Just a Few Lines of Code: the BabelNet API
- Joining Forces Pays Off: Multilingual Joint Word Sense Disambiguation
- BabelRelate! A Joint Multilingual Approach to Computing Semantic Relatedness
- Lexical semantic typologies from bilingual corpora - A framework
- Min-K%++: Improved Baseline for Pre-Training Data Detection from Large Language Models
- Benchmark Data Contamination of Large Language Models: A Survey
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Language Contamination Explains the Cross-lingual Capabilities of English Pretrained Models