Speech recognizers do far worse on children than on adults. Word error rates for young children can be 10 to 19 times higher, according to Large Language Models based ASR Error Correction for Child Conversations. Progress is now tracked on a handful of public corpora: MyST, CMU Kids, OGI Kids, CHILDES, and Jasmin-CGN. Many of these have been online for decades. The recognizers and the language-model "correctors" evaluated on them were pre-trained on web crawls. For adult benchmarks, it is already known that test sentences leak into those crawls. Evaluation of LLMs in Speech is Often Flawed found roughly two-thirds of the LibriSpeech dev and test sets inside The Pile. That paper also found the leak biased language models toward the memorized sentences, but only mildly affected final adult error rates, because clear adult audio pins the decoder down.
Our claim is that children are exactly the case where that reassurance fails. When the audio is ambiguous, because of high pitch, shifted formants, mispronunciations and false starts, the decoder leans on its text prior. If that prior has memorized the test sentence, it can output the correct text without having heard it. We propose a standardized child-speech evaluation protocol that reports this effect explicitly. The protocol attaches to every word error rate (WER, the fraction of words wrong) a measured leak gap: how much better the system does on sentences it could have memorized than on acoustically matched sentences it could not.
Core Claim
The benefit a model gets from memorized test text should grow as acoustic evidence weakens. If so, published child ASR gains, and especially language-model error-correction gains, are inflated most precisely for the hardest speakers (ages 4–7) and the weakest recognizers. String-overlap audits cannot measure this inflation. They cannot see inside closed models such as Whisper or GPT-4o-mini. And overlap is not effect: a leaked sentence only matters if the model uses it.
What is new is a behavioural measurement that needs no access to training data. On read child speech, we build counterfactual test items by splicing the same child's own recorded words into the sentence. This yields audio of a sentence that exists nowhere on the web, spoken in that child's voice. We then compare errors on original and counterfactual items.
Research Program
The program has three parts.
The first part is a conventional audit, done carefully. We search the open pre-training corpora named in the adult study (The Pile, Project Gutenberg, Wikipedia) for the transcripts and reading prompts of the corpora in the file. We reuse that paper's matching method of Jaccard similarity plus manual verification. Each test utterance is labelled leaked or unleaked in open corpora. This label is only a lower bound for closed models, which is why the next part exists.
The second part is the counterfactual splice test, which is the core contribution. Read-speech corpora reuse fixed prompt lists: CMU Kids, OGI Kids Scripted, and Jasmin-CGN, whose reading-mistake annotations are described in Improving Child Speech Recognition and Reading Mistake Detection by Using Prompts. For each prompt sentence, we use forced alignment to find a content word. We then replace it with a different word of matched frequency and phone length, taken from elsewhere in the same child's recordings. The result is a sentence that is fluent but has never been published, spoken in that child's voice.
Splicing introduces boundary artifacts. To cancel them, we create a matched "sham splice" for each item: the original word is replaced by a different recording of the same word by the same child. The leak gap for a system $s$ is then
$$G_s = \mathrm{WER}_s(\text{counterfactual splice}) - \mathrm{WER}_s(\text{sham splice}),$$
measured only on the spliced word and its immediate neighbours. Both conditions contain one splice, so artifacts cancel. Only one has text that could have been memorized.
A second signal comes from Jasmin-CGN's annotated reading mistakes. At positions where the child actually misread, we count how often the system outputs the canonical prompt word instead of what was said. We call this prompt regression. It matters doubly here, because reading assessment depends on catching those mistakes. Wieke Harmsen et al. in Can ASR generate valid measures of child reading fluency? show that ASR-derived fluency measures depend on correctly registering deletions and substitutions.
The third part is a controlled injection experiment, which establishes causality where the behavioural test can only show correlation. We take open correctors and decoders and continue pre-training them with child test transcripts injected at controlled exposure counts. The exposure-count design follows Detecting Benchmark Contamination Through Watermarking. The correctors include Flan-T5 fine-tuned on CHSER as in CHSER, and a small open language model used as an N-best rescorer. We inject one random half of the transcripts and hold the other half out.
We vary acoustic evidence strength along three axes:
- Natural axis: speaker age group, as in the ESPnet benchmark.
- Recognizer axis: Whisper versus a CTC WavLM recognizer, whose hypotheses are much noisier per Large Language Models based ASR Error Correction for Child Conversations.
- Synthetic dial: added noise and filtering drawn from Speech Robust Bench, applied to child audio rather than adult audio.
The prediction is that the injected-minus-held-out WER difference grows with the uninjected WER. The calibration question is whether the splice-test gap $G_s$ tracks that injected difference closely enough to serve as a proxy for closed models.
The deliverable is a set of evaluation recipes added to the existing ESPnet child benchmarks from Benchmarking Training Paradigms, Dataset Composition, and Model Scaling for Child ASR in ESPnet. For every system, the recipes report WER, leak gap, and prompt-regression rate, using a common text normalization across corpora. The common normalization matters because MyST Phase I and Phase II transcribe disfluencies differently. Without it, a leak gap could simply reflect annotation-convention mismatch, the same lesson Pyysalo drew for parsers in On the unification of syntactic annotations.
Baselines
The naive reference point is plain WER on the original static test sets, reported with no contamination accounting, as every child ASR paper in the file currently does. For example, the ESPnet benchmark reports 5.0% WER for fine-tuned Whisper-small on CMU Kids. Our protocol "beats" this reference if the leak gap is reliably nonzero for some systems, and if correcting for it changes a ranking or a claimed improvement. If every gap is within noise, that null is itself the useful finding: it would license the existing static numbers.
The strongest existing contamination method for speech is the overlap audit of Evaluation of LLMs in Speech is Often Flawed, which reported a −2.5 perplexity difference on leaked sentences. We run it on the same child test sets with the same matching thresholds. Its leaked/unleaked labels then serve as a competing predictor of the injected WER difference. The splice test beats it if, in the controlled injection experiment, $G_s$ explains substantially more of the variance in the true injected effect than overlap labels do. The splice test must also give usable estimates for closed models, where overlap labels are undefined.
The strongest systems whose gains are at stake set the effect sizes we must be able to resolve:
- CHSER Flan-T5 corrector: up to 28.5% relative WER reduction zero-shot, and 12.9% absolute on OGI Scripted.
- Llama 3.1-8B N-best correction of zero-shot Whisper on ADOS: 40.77% to 37.09%, per Large Language Models based ASR Error Correction for Child Conversations.
- GPT-4o-mini with hypothesis alignment on Jasmin-CGN: 5.1% WER from a 9.4% Whisper baseline, and mistake-detection F1 rising from 0.39 to 0.73.
All systems are run on identical audio, with the same decoding settings and beam width, and with no tuning on the counterfactual items. For injection runs, the injected and clean models share the same compute and token budget.
Two ablations isolate the key ingredient. The first drops the sham splice and compares counterfactual splices against unspliced originals; this shows how much of a naive gap is splice artifact. The second replaces frequency-matched substitute words with unmatched ones; this shows how much is a mere fluency effect rather than memorization.
Why not just…?
Why not just run n-gram decontamination on the training data? For open corpora we do, as one baseline. But Whisper's training data is described in the ESPnet study as a black box, and GPT-4o-mini's is unavailable, so overlap cannot be computed for the systems that matter most. Even where overlap is computable, the adult study shows it does not translate straightforwardly into WER effects. The watermarking paper further reports that simple canary strings often go unmemorized, so presence and use come apart. The injection experiment tests directly whether overlap labels predict the effect.
Why not just record fresh child speech with new prompts? This is the gold standard, and a real threat to the proposal's necessity. It is also slow, small, and constrained by children's voice-privacy concerns (see Children's Voice Privacy: First Steps and Emerging Challenges). And once published, a fresh set begins to leak, as LiveBench-style refreshing in text acknowledges. We treat a small fresh set as a validation target, if one can be collected with an existing partner. If the splice gap agrees with the fresh-versus-old gap, the cheap test can be rerun on any corpus indefinitely. If it disagrees, the splice test is falsified as a proxy.
Why not just use perplexity-based membership inference on the language model? Child prompts and child-directed text are short and simple, so they get low perplexity whether or not they were memorized. This is exactly the confound the frequency-matched substitution is designed to remove. Perplexity also says nothing about the acoustic interaction, which is our central hypothesis. We include perplexity as an additional predictor in the calibration regression, and report whether it adds anything beyond $G_s$.
Why not just evaluate on spontaneous speech like MyST, which has no fixed prompts? Spontaneous transcripts can also be published and scraped. More importantly, read speech is where the clinical and educational use cases (reading-fluency assessment) live, and where prompt regression does direct harm. We do report MyST results using the open-corpus audit and the injection experiment.
Decisive Checks
Three results would settle the proposal.
The first is the slope test. In the controlled injection experiment, does the injected-minus-held-out WER difference increase with acoustic difficulty? We check this across age groups, across Whisper versus CTC hypotheses, and across the noise dial. A flat slope means child contamination is no worse than adult contamination, and the central claim fails.
The second is calibration. Across injected models, does $G_s$ from the sham-controlled splice test correlate with the true injected effect better than overlap labels or perplexity do?
The third is the real-world readout. Applied to off-the-shelf Whisper and the published correctors, do leak gaps or prompt-regression rates on Jasmin-CGN mistake positions account for a meaningful share of the reported correction gains? The Jasmin gain from 9.4% to 5.1% is the most suspicious case, since the prompting paper itself notes Whisper may treat the reading text as a spelling reference.
We will publish the splice items, normalization rules, and recipes. Others can then report leak-adjusted child ASR numbers under identical conditions.
References
- Large Language Models based ASR Error Correction for Child Conversations
- Evaluation of LLMs in Speech is Often Flawed: Test Set Contamination in Large Language Models for Speech Recognition
- Improving Child Speech Recognition and Reading Mistake Detection by Using Prompts
- Can ASR generate valid measures of child reading fluency?
- Detecting Benchmark Contamination Through Watermarking
- CHSER: A Dataset and Case Study on Generative Speech Error Correction for Child ASR
- Speech Robust Bench: A Robustness Benchmark For Speech Recognition
- Benchmarking Training Paradigms, Dataset Composition, and Model Scaling for Child ASR in ESPnet
- On the unification of syntactic annotations under the Stanford dependency scheme: A case study on BioInfer and GENIA
- Children's Voice Privacy: First Steps and Emerging Challenges