Do Synthetic Multilingual QA Gains Survive a Clean Test?

Large language models are increasingly used as "teachers" that write training data for smaller models in languages where human labels are scarce. In multilingual question answering, the results look excellent. A 540-billion-parameter teacher shown only five examples per language produces data that beats translated training data on a standard benchmark. The catch is that the teacher has read most of the public web, very likely including the benchmark itself. If it has memorized benchmark questions and answers, possibly in another language, its "synthetic" data may quietly pass those answers to the student. The student's score would then reflect leaked test content rather than skill gained cheaply.

This note proposes to measure that leakage directly for extractive question answering, where a model must copy an answer span from a given passage. The main tool is a counterfactual-passage test. We edit the answer inside a benchmark passage and ask whether the model still outputs the original answer. A model that reads the passage gives the edited answer. A model repeating a memorized question-answer pair gives the old one. We pair this test with a small benchmark built from Wikipedia articles written after the teacher's training cutoff. Together they let us ask how much of the reported data efficiency of teacher-generated multilingual data is real. They also test a concrete fix: generating synthetic data only from passages the teacher cannot have memorized.

Core Claim

Our claim is that teacher-generated cross-lingual QA data can carry benchmark memorization to the student, including across languages. This inflation can be measured with no access to the teacher's pretraining data by combining two signals. The first is how often the student outputs the original answer on counterfactually edited benchmark passages, compared with matched non-benchmark passages. The second is how much of the student's gain on static benchmarks is lost on a fresh post-cutoff set. If the claim holds, part of the reported advantage of LLM-generated data over translate-train disappears on clean data. If it fails, we will have the first contamination-controlled evidence that these gains are genuine, which the field currently assumes but has not checked.

The motivation comes from three findings. QAmeleon prompt-tunes PaLM-540B on five examples per language, generates QA pairs from target-language Wikipedia, and trains mT5-XL. It did not check whether the teacher had seen TyDiQA or MLQA. Contamination Report for Multilingual Benchmarks finds that recent open models such as Llama-3.1-8B and Gemma-2-9B are contaminated on nearly every multilingual benchmark it audits. Data Contamination Can Cross Language Barriers shows that training on a French translation of MathQA lifts LLaMA3-8B's English accuracy from 42.01% to over 95%, invisibly to n-gram and perplexity checks.

Two papers constrain the design. Synthetic Data and the Rise of Spiky Intelligence shows that synthetic data concentrated near a benchmark inflates English math scores as much as verbatim leakage. It reports Inflation Scores up to 2.5, and 3.1 on surface-rewritten problems. We adopt its logic of comparing benchmark gains with capability-matched held-out gains, but apply it to multilingual extractive QA and to a teacher-to-student channel it does not study. Yao et al.'s Choice Confusion detector works only for multiple choice, so we design an analogue for span extraction.

Research Program

The work has three parts.

Counterfactual-passage test. For each TyDiQA-GoldP, MLQA, and XQuAD evaluation item, we replace the gold answer span and its coreferent mentions with a substitute of the same type: another date, a person of the same nationality, or a number of similar magnitude. We keep the question unchanged and the passage fluent. Native-speaker checks on a sample confirm grammaticality in morphologically rich languages, where TyDi QA (TyDi QA) warns that phenomena such as Finnish consonant gradation make naive substitution error-prone.

The memorized-answer rate is the fraction of edited items where the model outputs the original, now-absent answer. Models can also stick to an original answer because of general world knowledge, which has nothing to do with test leakage. To separate the two, we apply the identical editing procedure to matched control items. These use other paragraphs of the same Wikipedia articles, which are not in any benchmark, with questions written to the same guidelines. The leakage signal is the difference in memorized-answer rate between benchmark items and control items.

Fresh post-cutoff set. Native speakers write TyDiQA-GoldP-style questions over Wikipedia articles created after the teacher's training cutoff. We target a few hundred items per language across the TyDiQA languages, and include Hindi, Arabic, and Spanish so that MLQA-style cross-language queries can also be tested. This set is the ground truth against which the counterfactual test is validated. It will expire once crawled, which is exactly why the cheaper and reusable counterfactual test matters.

Generation-side decontamination. We regenerate synthetic data under three passage sources. The first is the same Wikipedia pool the benchmarks drew from, which mirrors current practice. The second is post-cutoff articles only. The third is benchmark-pool passages with counterfactual entity edits applied before generation. If the second or third source preserves most of the gain over translate-train, it becomes a practical recipe for leakage-safe synthetic data.

Baselines

The naive reference point is zero-shot transfer: the student fine-tuned only on English QA data and evaluated directly in target languages. QAmeleon reports this English-only baseline at 58.5% Exact Match (EM) on TyDiQA-GoldP. Every method is measured by how much of the gap above this floor it closes, on static benchmarks and on the fresh set.

The strongest existing method is QAmeleon-style teacher generation, which reports 70.2% EM on TyDiQA-GoldP (QAmeleon). PaLM-540B is closed, so we reimplement the pipeline with an open teacher (Qwen 2.5 72B, the generator used in XL-Instruct). The reimplementation keeps the same five examples per language, a matched number of synthetic pairs, and an mT5-XL student. We also include the question-generation pipeline of Synthetic Data Augmentation for Zero-Shot Cross-Lingual Question Answering, which reports MiniLM EM rising from 26.0 to 41.4 on MLQA and from 29.5 to 49.1 on XQuAD. Its generator is a small model trained on SQuAD, so it has a different contamination profile from a web-scale teacher.

Translate-train is the strongest competitor that does not depend on the teacher, so the teacher's memory cannot inflate it. QAmeleon reports it at 66.1% EM on TyDiQA-GoldP. It is the key comparison: the reported 4.1-point lead of teacher generation over translate-train is exactly the quantity we test for leakage.

All students share the same architecture, number of training steps, and hyperparameter search budget, with selection on a dev split that is never used for counterfactual analysis. Results are reported over three seeds.

The ablation that isolates our key ingredient is a teacher with planted contamination. We fine-tune a copy of the open teacher on TyDiQA-GoldP and MLQA test items, plus machine translations of them in the style of Yao et al., then run the identical generation pipeline. This gives a known positive case. The detector beats its competitors if students of the planted teacher show a clearly elevated benchmark-minus-control memorized-answer rate, and that elevation predicts the static-minus-fresh score gap across conditions better than the competing detectors do.

Why not just…?

Why not just filter the synthetic data for n-gram overlap with the test sets? For same-language verbatim copies this may catch a lot, so it is a real competitor. We will run it, including n-gram matching against machine translations of every test set into every language. Its weakness is paraphrase and translation. Yao et al. show that translated contamination evades n-gram and perplexity checks while inflating English MMLU by 5 to 10%, and Schuyler et al. show that benchmark-adjacent data inflates scores with no shared n-grams. The planted-teacher condition settles the question. If n-gram filtering removes the planted teacher's inflation as fully as post-cutoff generation does, we will report that the simple fix suffices.

Why not just evaluate on a fresh native-speaker set and skip the detector? The fresh set is the gold standard, and we build one. But it costs annotator time for every language and every evaluation cycle, and it stops being clean once crawled. The counterfactual test reuses existing benchmarks indefinitely. We will only claim it as a substitute if its leakage signal tracks the fresh-set gap across conditions. If it does not, the paper's conclusion becomes "build fresh sets," and that is still worth knowing.

Why not just run the black-box exchangeability test on the student? That test flags memorization of a dataset's canonical example order, at p < 0.0013 (Contamination Report for Multilingual Benchmarks). Synthetic data passed through a teacher does not preserve that order, so the test is blind to this channel by construction. It is also binary and cost about 362 hours on eight H100 GPUs for that paper's model suite. We will still run it on the teachers to report their baseline contamination status.

Why not just use translate-train, which has no teacher memory? If teacher generation's lead disappears on the fresh set, this is exactly the right conclusion, and the experiment is designed to reveal it. If the lead survives, especially with post-cutoff generation, the teacher adds something translation cannot, such as native phrasing and reduced translationese.

Proposed Experiments and Decisive Checks

The benchmarks are TyDiQA-GoldP, MLQA (both standard cross-lingual transfer and the setting where question and context are in different languages), and XQuAD, plus their counterfactual versions and the fresh set. We report EM, F1, the memorized-answer rate difference, and an inflation measure following Schuyler et al.: the student's gain over zero-shot on static benchmarks minus its gain on the fresh set.

The first decisive check validates the detector. Students of the planted teacher must show both a raised memorized-answer difference and a static-minus-fresh gap. Otherwise the counterfactual test is uninformative and we fall back on fresh sets alone.

The second check is the cross-lingual channel. XQuAD is a translation of SQuAD, so we test whether a teacher contaminated only on English SQuAD produces German or Hindi synthetic data that raises the student's memorized-answer rate on XQuAD in those languages. This would extend Yao et al.'s cross-language leakage from direct fine-tuning to synthetic distillation.

The third check is the substantive one. With an unmodified public teacher, does the lead of teacher generation over translate-train survive on the fresh set, and does generating from post-cutoff passages preserve the gains? A surviving lead supports the data-efficiency claims behind QAmeleon-style generation. A vanishing one means a share of reported multilingual synthetic-data gains has been borrowed from the test set.