Contamination Through the Sound Channel: Auditing Benchmark Leakage in Romanized and Homophonic Form

Benchmark contamination means a language model saw test questions during training, so its score reflects recall rather than skill. Current defenses look for leaked items in three ways. They search training text for exact matching word sequences, they search for rewordings with sentence embeddings and a judge model, or they look for unusually high model confidence on test items. All three assume the leaked copy is written the way the benchmark is written. That assumption breaks for languages people routinely type by sound. Hindi, Tamil and Thai are often written in the Latin alphabet on the web ("Hinglish"). Chinese can be typed as Pinyin or with same-sounding characters. Japanese can be written in kana instead of kanji. A Chinese benchmark question copied into Pinyin shares no characters with the original, yet it says exactly the same thing. We propose to measure whether leakage in this "sound-alike" form raises a model's score on the original benchmark. We also propose a cheap detector that catches such leakage by comparing text by pronunciation rather than spelling.

The second contribution borrows directly from evaluation work in transliteration and speech recognition. That field long ago stopped treating valid spelling variants as errors and began scoring them in a pronunciation space. Examples include character-level longest-common-subsequence scores in the transliteration shared tasks, tables of interchangeable same-sounding characters, kana–kanji lattices, and BLEU computed over phonetic transcriptions. We turn these scoring tools around and use them as matching keys for finding leaked data. If sound-alike exposure turns out not to inflate scores, that is also a useful result, because it tells auditors of multilingual benchmarks they can safely ignore this channel.

Core Claim

We make two claims that can be tested separately. First, exposing a model to romanized or homophonic copies of benchmark items raises its accuracy on the canonical-script items. This inflation escapes raw n-gram filters and escapes likelihood-based detectors applied to the canonical text. Second, matching benchmark items against corpus text after mapping both to a coarse pronunciation key recovers most of this leakage. It does so at n-gram cost and without a judge model. It also beats the obvious shortcut of transliterating the benchmark once with a standard romanizer, because informal web respellings are many-to-many rather than a single canonical romanization.

No paper in the file tests either claim. Rethinking Benchmark and Contamination for Language Models with Rephrased Samples established that rewritten test items evade n-gram checks and still inflate scores; it reports that fine-tuning Llama-2-13B on rephrased MMLU items raised its score from about 55 to about 86. Its rewrites are semantic ones: synonyms, reordering, translation into another language, and code translation. It never tests rewrites that keep the pronunciation of the original, in the same language. When Is Benchmark Contamination Detectable? ranks how much detectable signal survives different kinds of exposure. It finds verbatim exposure leaves the most signal, followed by paraphrase, then token shuffling, then answer-only exposure. Sound-alike exposure is absent from that ranking, and placing it there is our first measurement. Contamination Report for Multilingual Benchmarks detects multilingual leakage by testing whether a model prefers the benchmark's original item order. That test only fires if the benchmark file was ingested in its original form and order. It therefore cannot see scattered, respelled copies of the kind that appear in forum posts or tutorials.

Research Program

The work has three parts.

The first part builds realistic variants. For Chinese, we generate full tone-free Pinyin renderings. We also generate partial homophone substitutions drawn from same-Pinyin character sets, following the Alternating Character Table idea of English-to-Chinese Transliteration with Phonetic Auxiliary Task. For Japanese, we apply the kanji-to-hiragana and hiragana-to-katakana conversions of Lenient Evaluation of Japanese Speech Recognition. That paper reports that human raters judged 95.4% of such variants plausible, which is our justification that these variants look like real text. For Tamil and Thai, we produce informal Latin-script respellings. We sample from a spelling-variant model rather than a single romanizer, because CoMix documents the spread of informal spellings ("kis", "kys", "kise" for one word). Native speakers will check a sample of variants in each language, and we will report the acceptance rate.

The second part is a controlled injection study in the style of When Is Benchmark Contamination Detectable?. The base model is Aya-23-8B, which passed the item-order test on XCOPA and PAWS-X in the multilingual report and is therefore a relatively clean starting point. We continue training it on a background multilingual corpus. Into that corpus we inject a random half of the benchmark items at a controlled fraction $\alpha$, in one of four forms: verbatim, semantic paraphrase, sound-alike variant, or none. The other half of the items is held out as a matched clean control. This design gives us the unexposed reference set that the power analysis in that paper requires.

The third part is the detector. Let $\phi$ map a string to a coarse pronunciation key. For Chinese, $\phi$ is tone-free Pinyin. For Japanese, it is a hiragana reading. For Indic and Thai text, in either native or Latin script, it is a normalized phoneme string that collapses vowel length and doubled letters. We score a benchmark item $r$ against a corpus window $c$ with

$$ s(c, r) = F_{\text{LCS}}\big(\phi(c), \phi(r)\big). $$

Here $F_{\text{LCS}}$ is the character-level F-score based on the longest common subsequence, taken from NEWS 2018 Whitepaper. The number of windows scored is kept manageable by first indexing n-grams of $\phi$-keys to retrieve candidates, so the whole pipeline scans a corpus at n-gram-filter cost.

Baselines

The naive reference is raw n-gram overlap on the original script, which is standard decontamination practice. Yang et al. showed it misses rephrased items. It should miss sound-alike items entirely, and measuring that confirms the injection works.

The strongest existing corpus-side method is the LLM Decontaminator from Rethinking Benchmark and Contamination. It retrieves the top-$K$ corpus windows by embedding similarity, then asks a strong judge model whether each is a rewritten test item. It reports F1 between 0.94 and 1.0 on semantically rephrased samples. There is no reported result on Lacuna for romanized or homophonic contamination. We run it on the same injected corpus with the same number of candidates per item as our detector. Because full-corpus judging is expensive, we run both methods on an identical corpus subsample, which keeps the comparison fair. Our detector beats it if it matches the Decontaminator's F1 on sound-alike items within noise at a small fraction of the compute cost. It also beats it if it exceeds the Decontaminator's F1 on the informal Tamil and Thai respellings, where we expect embeddings to break down.

The strongest model-side baseline is Min-K%++ from Min-K%++. This method scores how far each token's log-probability stands out above the model's other token choices at that position, and needs no reference model. It reports gains of 6.2% to 10.5% AUROC over Min-K% on WikiMIA. We apply it to the canonical-script items of the injected model and report AUROC for exposed versus control items, separately for each exposure form.

The key ablation is to use a single standard romanizer in place of $\phi$ and then run exact n-gram matching. This isolates whether the gain comes from pronunciation-level collapsing and the tolerant LCS score, or merely from switching scripts.

Why not just…?

Why not just romanize the benchmark with a standard tool and run n-gram matching? This is the most serious threat. For Pinyin, whose romanization is close to deterministic, it may well match our detector. The ablation settles the question for each language. We expect the gap to appear only for informal Indic and Thai respellings, where CoMix shows many spellings per word. If there is no gap even there, we will report the romanize-then-match method as the recommended audit. The paper then becomes a measurement study about contamination through the sound channel rather than a detector paper.

Why not just use embeddings plus an LLM judge? Yang et al. already needed the judge because embedding similarity alone flags topically related clean text. Running a judge over web-scale corpora is impractical. Yang et al. also found 12.8% of CodeAlpaca to be rephrased HumanEval, which shows that teacher models can themselves carry benchmark leakage, so a judge is not a neutral arbiter. Our head-to-head comparison on the subsample measures whether embeddings retrieve informal respellings at all.

Why not just run Min-K%++ or other model-side detectors and skip corpus access? These detectors look for sharp peaks in token likelihood on the exact token sequence the model was trained on. Sound-alike exposure trains the model on different tokens. The inflation, if it exists, would therefore show up on canonical items without that peak. The per-form AUROC measures this directly. If Min-K%++ detects sound-alike exposure as well as it detects paraphrase, corpus-side phonetic matching is unnecessary for models whose weights we can query, and we will say so.

Why not just use dynamic benchmarks? DyVal generates items procedurally, but only in formal domains such as arithmetic, logic and graph problems. It cannot replace XCOPA-style commonsense or PAWS-X-style paraphrase judgments in natural language.

Proposed Experiments

The benchmarks are XCOPA (Chinese, Tamil and Thai splits) and PAWS-X (Chinese and Japanese splits). The primary behavioral quantity is score inflation, defined as accuracy on exposed items minus accuracy on matched control items, measured for each exposure form and each $\alpha$. XCOPA reports cross-lingual accuracy of roughly 60–70%, which leaves room for inflation to show. Detection is scored as F1 and recall on injected corpus windows for the corpus-side methods, and as AUROC for Min-K%++. We also estimate the separability $\rho$ for sound-alike exposure and place it in the ranking of exposure types from Shihab et al.

The first decisive check is whether sound-alike exposure produces inflation significantly above zero on canonical items. If it does not, claim one is false. We would publish that negative result and limit the detector's use to benchmarks evaluated in romanized form. The second is whether $\phi$-key matching beats the single-romanizer ablation on informal respellings. The third is whether both n-gram filtering and Min-K%++ fail on items where inflation is present.

Finally, we will run the matcher over a public multilingual web corpus to look for sound-alike copies of XCOPA and PAWS-X items. We will verify hits by hand and report any found. We will not presume that leaks of this kind exist in the wild.