Speech benchmarks such as LibriSpeech are badly leaked. Their transcripts sit inside the text corpora that train the language models now built into speech recognizers and audio-language models (systems that take audio in and produce text answers). Existing audits check whether the transcript appears in a known text corpus. They cannot say whether a given audio model actually exploits that leak, and they fail when the training data is private. We propose a behavioral audit that needs no training data and uses a tool hearing science has relied on for decades: the cochlear-implant simulation. This is a vocoder that keeps a sound's slow loudness envelope but removes fine frequency detail by collapsing the spectrum into a few channels. It can also shift those channels along the frequency axis to mimic an electrode placed at the wrong position along the cochlea's frequency map (its "tonotopic" map, where each place responds best to one frequency).
The idea fits in two sentences. As we progressively remove acoustic evidence, an honest model's transcription accuracy should fall with intelligibility. A model whose language component has memorized the benchmark sentence should keep producing the right words long after the audio stops supporting them, especially once a short clean prefix tells it which sentence it is hearing. Measuring how accuracy decays along calibrated degradation axes therefore separates three cases: clean models, models that memorized the transcripts, and models that memorized the audio itself.
Core Claim
The closest analogue is Contamination Detection for VLMs Using Multi‑Modal Semantic Perturbations. It edits benchmark images with diffusion models while keeping the task fixed, and it finds that contaminated vision-language models collapse on the edited versions (Δ down to −45% on MMStar) while clean models improve (Δ = +31.5%). Ported naively to audio, this would mean "perturb the audio and look for a drop." We argue that for speech the sign is often reversed, and that this reversal is what makes a useful detector.
In speech, the dominant leak is textual. Evaluation of LLMs in Speech is Often Flawed reports that about two-thirds of LibriSpeech dev/test and about one-third of Common Voice English dev/test appear in The Pile. When audio evidence is degraded, a model with a memorized prior fills the gap from memory, so contamination shows up as unexpected robustness rather than fragility. Memorization of the audio itself, the case the vision analogue targets, should instead show up as fragility that is specific to frequency shift: a fingerprint tied to exact spectral placement should break when the channels move, even at degradation levels where humans and clean models still recover the words.
Our claim is that the shape of the accuracy-versus-degradation curve, read across a channel-count axis and a tonotopic-shift axis, gives a per-utterance contamination signal. That signal should outperform transcript likelihood and should survive the reinforcement-learning post-training that hides loss-based signals.
The statistic is simple. Let $a_i(d)$ be word accuracy on utterance $i$ at degradation level $d$, after a clean prefix covering the first fraction $p$ of the utterance. We fit a reference curve $\hat a(d \mid a_i(0), \ell_i)$ on a known-clean reference set as a function of clean accuracy and utterance length $\ell_i$. The memorization lift is
$$S_i = \frac{1}{D}\sum_{d=1}^{D}\big(a_i(d) - \hat a(d \mid a_i(0), \ell_i)\big),$$
where $D$ is the number of degradation levels. A positive $S_i$ indicates the text prior is doing work the audio cannot justify. A strongly negative $S_i$ that appears only on the shift axis indicates an acoustic fingerprint.
Research Program
Degradation axes. We build the degradation family from the cochlear-implant simulation literature. Integration of acoustic and electric hearing is better in the same ear than across ears used a vocoder at 1.2–8.0 kHz combined with low-passed acoustic input at 0.1–0.6 kHz. It showed that fusing the two works well when their frequency maps align (a 35.1-point benefit over vocoder-only) and degrades sharply under tonotopic mismatch. That paper supplies our method and motivates the two axes. Channel count removes spectral detail but keeps timing. Basal shift preserves the information while misplacing it, which human listeners partly adapt to. We use only the transformations, not any claim about the slope of the mismatch effect; the file reports no number for it. As a third, non-tonotopic arm we add speech-shaped noise at matched levels. This arm is essential: it tests whether the frequency-map structure matters at all.
Ground-truth contamination. We create known contamination in an open audio-language model such as Qwen2-Audio, one of the systems evaluated in AHELM, and in a Whisper-style recognizer fused with a Pythia language model, following the controlled-dose design of the speech contamination paper. There are three regimes on disjoint halves of LibriSpeech test-clean/test-other and Common Voice English. The first is clean. The second is text-only: the language component is fine-tuned on test transcripts. The third is audio-text: the full model is fine-tuned on test audio-transcript pairs. Each regime is run at several exposure doses.
Natural experiment. The leaked-versus-non-leaked labels from the speech contamination study give a second, uncontrolled test on Pythia models that were never deliberately contaminated.
Post-training stress test. We apply one epoch of clean GRPO, the reinforcement-learning recipe that On The Fragility of Benchmark Contamination Detection in Reasoning Models shows compresses the loss gap between seen and unseen items, to the contaminated audio-language models. We then re-run every detector. Our prediction is that clipping compresses likelihood differences but does not erase the model's tendency to complete memorized text from weak input. We treat this as a falsifiable claim, not an assumption.
Baselines
Naive reference. Transcript n-gram or Jaccard overlap against The Pile, the method of the speech contamination paper. It is the obvious first check and is exact when the corpus is available. We run it on the same utterances. On our controlled models it is trivially perfect for text exposure, because we know the corpus, and undefined for private data. We therefore report it as an upper reference for the text regime and as unavailable in the private-data setting the audit targets. Beating it means matching its ranking without corpus access.
Strongest likelihood-based detector. Min-K% and reference-model likelihood ratio (LiRA) scores computed on the transcript tokens, conditioned on the clean audio, as evaluated by Tang et al. On text reasoning benchmarks they report LiRA at 89.1% AUROC (area under the curve separating members from non-members) after SFT contamination. AUROC drops by about 14% after one GRPO epoch, and reference-free detectors fall to about 50%. The speech paper reports a −2.5 perplexity shift on leaked sentences, but in punctuated ASR only 1.603 versus 1.607, a near-null signal. We found no reported result on Lacuna for per-utterance contamination-detection AUROC on any audio benchmark.
All detectors receive the same models, utterances, and reference model. Where a detector needs a calibration set, it gets the same held-out clean split as our reference curve. Beating this baseline means a higher AUROC than LiRA before GRPO, together with a smaller drop than LiRA after GRPO.
Representation-geometry detector. The Eigenspectrum Concentration Index from A Narrowing Geometry in Contaminated Reasoning. It found that leaked inputs collapse activations in feed-forward up-projections onto a few dominant directions. We compute it on the language decoder and the audio encoder. It is white-box, so it serves as an upper competitor rather than an equal one. We report whether the behavioral curve matches it.
Re-voicing perturbation. The direct port of the vision method: re-synthesize each transcript with text-to-speech and measure the accuracy change. It should catch audio memorization but by construction cannot see text memorization.
Key ablation. Speech-shaped noise at matched intelligibility, in place of the vocoder and shift axes. This isolates whether tonotopic structure, rather than evidence removal in general, adds anything.
Why not just…?
Why not just check the transcripts against the pretraining corpus? When the corpus is public, this answers whether the text leaked. It cannot answer whether a deployed model uses the leak, and the speech paper's own downstream perplexities of 1.603 versus 1.607 show leakage and exploitation can come apart. For closed systems like those ranked in AHELM there is no corpus to check. Our natural-experiment arm tests directly whether $S_i$ recovers the corpus-overlap labels without seeing the corpus.
Why not just use likelihood or Min-K% on the clean audio? Tang et al. show these signals shrink under clipped policy-gradient post-training, and the speech paper's perplexity gaps are tiny. Our GRPO arm is designed to show whether the degradation curve degrades less. If it collapses just as badly, the proposal's main selling point fails, and we will report that.
Why not just add white noise instead of borrowing cochlear-implant tonotopy? This is the real threat. For the text-memorization regime, any evidence removal may produce the same plateau, and noise may work as well as the vocoder. The matched-intelligibility ablation decides it. If noise equals channel reduction for text contamination, we say so. The tonotopic component then earns its place only if the shift axis, which preserves envelope and phonetic content but moves frequency placement, separates audio-memorized from clean models better than noise or re-voicing does.
Why not just re-voice the benchmark with text-to-speech, as was done for images? Re-voicing keeps the words, so a model that memorized the text is unaffected and looks clean. That is the mirror image of the regime our curve targets. We run re-voicing as a baseline and expect it to win on audio-text contamination and lose on text-only contamination.
Proposed Experiments
The primary metric is per-utterance AUROC for member versus non-member within each controlled regime. It is reported at each exposure dose, before and after GRPO, with set-level AUROC aggregated over 100-utterance subsets as a secondary measure. Clean-accuracy-matched reference curves are fit on a separate non-member split so that easy utterances are not mistaken for memorized ones.
Three checks decide the proposal. First, on text-only contamination, $S_i$ should be positive and should exceed transcript-LiRA AUROC at low doses, where likelihood gaps are small. Second, on audio-text contamination, the shift axis should drive accuracy below the reference curve while noise at matched intelligibility should not. If both behave the same, the tonotopic claim is refuted. Third, after one clean GRPO epoch, the AUROC loss for $S_i$ should be clearly smaller than the roughly 14% loss Tang et al. report for LiRA.
As a final application, we run the black-box audit on AHELM's open speech subsets for the publicly accessible systems. We report which models show prior-driven plateaus on LibriSpeech's leaked sentences relative to its non-leaked ones, and present this as evidence about how far to trust those leaderboard rows, not as an accusation.
References
- Contamination Detection for VLMs Using Multi‑Modal Semantic Perturbations
- Evaluation of LLMs in Speech is Often Flawed: Test Set Contamination in Large Language Models for Speech Recognition
- Integration of acoustic and electric hearing is better in the same ear than across ears
- AHELM: A Holistic Evaluation of Audio-Language Models
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- A Narrowing Geometry in Contaminated Reasoning