Does Benchmark Memorization Survive Distillation? Tracing Contamination from Cross-Encoder Teachers into Multilingual Hybrid Retrievers

Most fast multilingual search systems are trained by imitation. A slow, accurate model that reads the query and passage together (a cross-encoder) scores many query-passage pairs. A fast model that encodes queries and passages separately is then trained to reproduce those scores. The fast model can be a dense vector retriever, a learned sparse retriever that assigns weights to vocabulary terms, or a hybrid of the two. Headline results for such students on public multilingual benchmarks such as MIRACL are usually read as evidence of zero-shot cross-lingual transfer, meaning ranking skill learned mostly in English that carries over to other languages. Contamination research shows why that reading is fragile. Training on a translated copy of a benchmark inflates scores on the original while leaving no surface trace (Data Contamination Can Cross Language Barriers), and modern multilingual foundation models are routinely contaminated on public multilingual test sets (Contamination Report for Multilingual Benchmarks). No one has asked whether a teacher's memorized knowledge of a test set is passed down through its soft scores, or whether the student starts clean.

This note proposes a controlled study of that question, together with a detection test built for retrieval. We deliberately contaminate a cross-encoder teacher, in measured doses and possibly through a different language, with the test query-passage pairs of half of a benchmark's queries. We then distill it into dense, sparse, and hybrid students and measure how much score inflation reaches each student and through which channel. We use the resulting dose-response curves to turn a behavioural test into an estimate of inflation in nDCG points, rather than the yes/no verdict current tests give. Finally, we apply the calibrated test to released multilingual retrievers.

Core Claim

A student can inherit a teacher's memorization only through inputs the teacher actually scores. We therefore predict that inflation in the distilled student depends on the product of two quantities. The first is how contaminated the teacher is. The second is how much the distillation candidate pool overlaps with the evaluation corpus's judged positives. This overlap is common in practice, because MIRACL's training queries share its Wikipedia corpus (Making a MIRACL).

We further predict that contamination is not spread evenly across a hybrid retriever. A learned sparse head, trained to match teacher margins, can store memorized preferences as unusually heavy expansion terms on specific test passages. A fixed-size dense vector, whose limited capacity is a known weakness (Sparse, Dense, and Attentional Representations for Text Retrieval), may partly filter those preferences out.

The new contribution is not the observation that contamination exists. It is a measured account of whether distillation launders, amplifies, or filters contamination, and which channel of a hybrid retriever carries it. It comes with a detector whose output is an inflation estimate, validated against freshly written queries.

Relation to Prior Work

Three distillation systems set the practice we audit. Distillation for Multilingual Information Retrieval (Multilingual Translate-Distill) scores machine-translated MS MARCO passages with a MonoT5-based teacher and trains ColBERT-X, a model that keeps one vector per token. MILCO maps 39+ languages into an English sparse vocabulary. It distills from an English cross-encoder teacher, SPLADE-v3, and adds a source-language "echo" view for rare entities. Cross-lingual Knowledge Transfer via Distillation for Multilingual Information Retrieval fuses BM25, mDPR, mColBERT and an mMARCO-trained cross-encoder on MIRACL. All three treat benchmark gains as genuine transfer, and none audits the teacher. We reuse their recipes as students, and treat their reported numbers as the claims under test.

Two works motivate the study. Data Contamination Can Cross Language Barriers injects translated benchmarks into LLMs, which raises LLaMA3-8B on MathQA from 42.01% to over 95%. It detects this with Choice Confusion, which replaces the wrong answer options with correct answers from other items, and it shows n-gram and perplexity filters miss the effect. It studies only multiple-choice LLM tasks, not ranking, and not distillation. Contamination Report for Multilingual Benchmarks detects leakage by checking whether a model prefers the canonical order of test items over random permutations. Its output is binary, and it relies on the model generating text in sequence. A cross-encoder scores pairs independently, so there is no ordering signal to exploit.

We borrow Yao et al.'s injection design and the logic of Choice Confusion, and rebuild the probe for ranking. Language Contamination Explains the Cross-lingual Capabilities of English Pretrained Models constrains how we interpret results. Because pre-training leakage already drives apparent transfer, our experiments must measure inflation relative to an identically built clean arm rather than against zero.

Naver Labs Europe (SPLADE) @ TREC DL 2021 found that 26.7% of labeled positives in MS MARCO v2 were false positives. That finding warns that any probe keyed to judged passages will also reflect label noise, which our design must control.

Research Program

The first stage builds the contaminated teachers. We start from a multilingual MiniLM cross-encoder trained on mMARCO, following Huang et al. We split MIRACL test queries per language at random into an injected half and a control half. We then continue training the teacher on the injected half's judged pairs at several doses, measured in epochs over the leaked pairs mixed into mMARCO. We use three injection routes. The first is same-language. The second is translated into English. The third is translated into a second target language, which reproduces Yao et al.'s cross-language route. A clean arm receives the same number of extra training steps on matched mMARCO pairs.

The second stage distills each teacher with Margin MSE, which trains the student to match the teacher's score difference between a positive and a negative passage, as in the SPLADE work. We train three students: a dense bi-encoder, a SPLADE-style learned sparse model, and their score interpolation. The key manipulated variable is whether the distillation candidate pool includes MIRACL corpus passages. We compare three conditions: mMARCO-only pools, pools built from MIRACL training queries, and pools of synthetic queries generated over the MIRACL corpus.

We define inflation as the difference-in-differences of nDCG@10. It is the contaminated arm's gap between injected and control queries, minus the clean arm's gap between the same two sets. Because query assignment is random, this quantity isolates memorization from any overall skill the extra training adds.

The third stage builds the retrieval probe, which we call a counterpart swap. For each test query we build a counterpart of its judged positive that keeps the content but changes the surface form, using two methods. The first is back-translation of the passage. The second is the aligned passage from another language's Wikipedia edition, translated into the query language. A memorizing model prefers the original over its counterpart more than a clean model does. We run the same swap on the query side with paraphrased queries.

The per-query preference gap, contaminated minus clean reference, is the raw detection statistic. Fitting that statistic against measured inflation across the controlled doses gives a calibration curve. The curve converts a probe reading on an unknown model into an estimated inflation in nDCG points. To address label noise, we restrict calibration to positives that native-speaker re-assessment confirms, drawing on the Lassance audit's lesson.

Channel attribution comes from scoring the sparse and dense components of the hybrid separately. For sparse students, we also check whether the heaviest expansion terms on injected-query positives are unusually specific to the leaked query, compared with the same passages under the clean arm.

Baselines

The naive reference point is untrained BM25 and the BM25 plus mDPR interpolation. Making a MIRACL reports average nDCG@10 of 0.393 and 0.578 respectively, with Recall@100 of 0.889 for the hybrid. Neither involves a teacher, so any inflation they show on injected versus control queries must be zero up to noise. That makes them the sanity floor for the difference-in-differences measure. We run them on the identical query split.

The strongest existing multilingual system in the file is the fusion of hybrid retrieval, mColBERT and cross-encoder from Huang et al. It reports average development-set nDCG@10 of 0.764 on MIRACL. It is the strongest competitor for the claim that "distilled multilingual retrieval really is this good," because its cross-encoder is exactly the kind of mMARCO-trained teacher we audit. We rerun its cross-encoder under our clean and contaminated arms.

For sparse students, MILCO is the strongest recipe. The file reports its zero-shot MKQA Recall@100 of 76.6 but no exact MIRACL number; no reported result was found on Lacuna for MILCO's MIRACL nDCG@10. We reproduce its distillation stage with our teachers under a matched training budget: same distillation queries, same negatives, same steps, and the same hyperparameter search for every arm.

For detection, the baselines are the methods practitioners actually use. These are n-gram overlap between teacher training data and the test set, a perplexity check with a multilingual language model, the order-exchangeability test adapted by scoring test pairs in canonical versus permuted batches, and Choice Confusion adapted to ranking. The adaptation replaces a query's negatives with other queries' positives, so a clean ranker should improve.

The probe beats these if it separates contaminated from clean teachers and students at matched false-positive rate, where n-gram and perplexity checks fail on the translated routes. It must also estimate inflation within a stated error on held-out doses and languages. The central ablation removes the carrier. With mMARCO-only distillation pools, our claim predicts student inflation near zero even from a heavily contaminated teacher. If inflation stays high without the carrier, the mechanism we propose is wrong.

Why not just…?

Why not just decontaminate the teacher's training data with n-gram filtering? Yao et al. show that translated contamination produces zero n-gram overlap and evades perplexity checks while inflating MMLU by 5-10%. Our translated-injection arms test directly whether the same holds for cross-encoders. If n-gram filtering does catch our injections, that is a useful negative result and we will report it. It still cannot audit released teachers whose training data is unpublished.

Why not just evaluate on freshly written queries? This is the real threat, and it is the gold standard. It is also why we write fresh queries for a few languages, to validate the probe. But fresh queries cost native annotators for every language and every release. They also say nothing about which existing published numbers were inflated, and they go stale the moment they are released. The probe is worth building only if its inflation estimates agree with fresh-query gaps. We treat disagreement as falsifying the probe rather than as something to explain away.

Why not just run the black-box exchangeability test on the teacher? The exchangeability test works because a generative model assigns higher likelihood to the canonical order of text it has seen. A cross-encoder scores each pair independently and has no likelihood over sequences, so the signal the test relies on is structurally absent. We include the adapted version as a baseline to show this concretely rather than assert it.

Why not just skip distillation and deploy the hybrid interpolation? For this question, removing the teacher removes the thing being audited. It does not show whether reported distillation gains are real. The interpolation serves as our zero-inflation floor, and the gap between it and distilled students on control queries is the honest transfer gain we aim to report.

Proposed Experiments and Decisive Checks

The main experiment covers four arms: clean teacher plus three injection routes, each at three doses. It crosses three students with three distillation pools on MIRACL, using the injected/control split in every language. We report nDCG@10 and Recall@100, and inflation with bootstrap confidence intervals over queries. A secondary cross-language arm injects via French or German and via Chinese or Japanese. It tests whether Yao et al.'s finding, that European languages inflate more, carries over to ranking.

Validation uses native-speaker queries over the MIRACL corpus in three languages chosen to span script and resource level. One of them is Yoruba, one of the benchmark's two surprise languages. We compare the probe's estimated inflation for released teachers and students against their measured public-versus-fresh gap relative to a clean reference.

Three results would each be decisive on their own. First, if inflation vanishes without a shared-corpus carrier but appears with one, distillation passes contamination through a specific, avoidable channel, and pool construction becomes a practical fix. Second, if sparse students show larger inflation than dense ones and carry query-specific expansion terms on leaked positives, the hybrid's lexical side is not the "safe" channel it is usually assumed to be. Third, if the calibrated probe estimates fresh-query gaps within its stated error on released models, the field gains an inflation estimate where it currently has only binary tests. If none of these holds, and inflation is small everywhere, that is the reassuring answer for published multilingual distillation results. It is itself worth establishing.