TL;DR
Popular speech recognition evaluation datasets like LibriSpeech and Common Voice are heavily contaminated within public LLM pretraining corpora like The Pile. While this data leakage doesn't drastically alter downstream Word Error Rates (WER), it significantly skews text generation probabilities, masking true model generalization and biasing evaluations.
What Problem Does This Paper Address?
As Large Language Models (LLMs) dominate natural language processing, speech researchers have increasingly integrated them into Automatic Speech Recognition (ASR) pipelines, speech translation, and error correction systems. To evaluate these new LLM-based speech systems, researchers typically turn to standard benchmarks like LibriSpeech and Common Voice.
However, there is a catch: LibriSpeech is built from Project Gutenberg audiobooks, and Common Voice relies heavily on text from Wikipedia. Because Project Gutenberg and Wikipedia are foundational text sources for training almost all major LLMs, a massive conflict of interest arises. The evaluation data has likely already been seen by the LLM during its pretraining phase. This paper investigates the exact extent of this test set contamination and explores how it distorts the evaluation of speech recognition systems.
Methods
The authors tackled the problem using a three-pronged approach:
- Contamination Analysis: They quantified data leakage in a popular pretraining corpus, The Pile. For LibriSpeech, they used the MinHash Locality-Sensitive Hashing (LSH) algorithm to identify duplicate books. For Common Voice, they leveraged the Wikipedia Search API to cross-reference evaluation utterances with Wikipedia articles included in The Pile.
- Controlled Pretraining: To isolate the exact impact of contamination, the authors trained several billion-parameter LLMs (1.4B, 2.8B, and 6.9B parameters) from scratch. They intentionally contaminated some variants by mixing LibriSpeech evaluation books into the pretraining data while keeping other variants completely clean.
- Downstream ASR Integration: They built LLM-based ASR systems by connecting a frozen speech encoder (WavLM Large) to their custom LLMs via a small, trainable projector. They tested these systems across two paradigms: punctuated ASR (predicting mixed-case text with punctuation) and regular ASR (predicting lowercase text without punctuation).
Datasets
- LibriSpeech: Used for evaluation (dev-clean, dev-other, test-clean, test-other) and ASR training (960-hour set).
- Common Voice: English dev and test sets (version 20.0) were analyzed for leakage.
- The Pile: Used as the baseline public LLM pretraining corpus to check for real-world contamination.
Key Insights and Ideas
- The Perplexity Illusion: Simply looking at the perplexity of an off-the-shelf pretrained LLM cannot reliably expose contamination. As shown in Figure 1, leaked sentences are inherently more predictable by nature (even to an uncontaminated n-gram model).
- Scale Exacerbates Memorization: When comparing clean vs. contaminated models trained from scratch, larger models (6.9B) exhibited a much larger drop in perplexity on leaked data than smaller models (1.4B). This aligns with the understanding that larger models possess a higher capacity for memorizing training data.
- WER Hides the Bias: Surprisingly, downstream metrics like Word Error Rate (WER) or Character Error Rate (CER) barely change when using a contaminated LLM. This happens because beam search decoding only cares about the relative rank ordering of words. Even if contamination makes a sentence significantly more probable to the LLM, it might have already been the top choice, leaving the final transcribed text unchanged.
- Degradation on Unseen Data: Contamination is a double-edged sword. While it artificially inflates the model's likelihood of outputting leaked sentences, it simultaneously reduces the probability of generating non-leaked, genuinely unseen sentences.
Key Results
- Massive Leakage Found: Nearly two-thirds (61.8%) of the LibriSpeech evaluation utterances and one-third (31.7%) of the Common Voice English evaluation utterances are present verbatim in The Pile's pretraining data.
- Clear Probability Shifts: In controlled settings, contaminated LLMs consistently assigned higher probabilities (lower perplexities) to leaked sentences. For instance, a 1.4B parameter model showed a distinct perplexity improvement on leaked text, an effect that did not disappear even when the total volume of clean pretraining data was quadrupled.
- ASR Perplexity Skew: For the 6.9B punctuated ASR system, using a contaminated LLM significantly lowered the perplexity conditioned on audio for leaked test sets (e.g., dropping from 1.610 to 1.603 on
dev-clean), while slightly degrading performance on non-leaked text.
Limitations and Future Work
While this paper clearly demonstrates that test set contamination introduces systematic biases, it has a few limitations:
- Scope of Models: The models trained from scratch were capped at 6.9B parameters and trained on up to 60B tokens due to compute constraints—much smaller than production-grade LLMs (like Llama 3 or Mistral), which are trained on trillions of tokens.
- Language Constraint: The data analysis and experiments were strictly limited to English datasets.
The authors emphasize that as LLMs become deeply embedded in speech applications (like speech translation and dialog systems), the community must move away from contaminated legacy benchmarks and establish strictly held-out, genuinely unseen evaluation sets to ensure scientific rigor.
Summary updated