Text-to-speech (TTS) systems, models that turn written text into spoken audio, are still mostly scored on test sets drawn from decades-old public-domain audiobooks: LibriTTS, and LJSpeech before it. Large speech models are now trained on web-scale audio that very likely contains those same books, readers and sentences. A good score on these test sets may therefore mean one of three things. The model may be reading sentences it has already heard. It may be cloning speakers it has already met. Or it may never have been asked to pronounce anything newer than the Victorian novels it trained on. Text language models went through the same reckoning: contaminated benchmarks overstated reasoning, and time-stamped benchmarks such as LiveBench and RealTime QA became the fix. Speech synthesis has had no such reckoning.
This note proposes a controlled study and a dated evaluation protocol for TTS. The protocol pulls each month's test sentences from text published after a model's training cutoff and uses speaker prompts that were recorded fresh. Deliberate contamination experiments then measure how much of a standard test score comes from three sources: memorized text, memorized voices, and the absence of newly coined words and names. The protocol also treats the scoring tools themselves as possibly contaminated. The speech recognizer used to measure intelligibility may have memorized LibriSpeech too. The restoration model used to clean new audio may invent phonemes.
Core Claim
We claim that the standard LibriTTS-style evaluation overstates how intelligible and how speaker-faithful a TTS model is on present-day text. We predict that the largest share of the overstatement does not come from verbatim sentence memorization. It comes from two quieter sources. The first is lexical obsolescence: post-cutoff names and coinages the model has never seen, which is the speech analogue of the stale knowledge studied by Mitigating Temporal Misalignment by Discarding Outdated Facts. The second is a contaminated evaluator: a recognizer whose language prior "repairs" mispronounced but familiar book text.
What is new here is the factorization. Controlled training-set injection, dated text slices and a prior-free evaluator together attribute a score gap to specific causes, and each cause can be tested and falsified separately. A single "fresh test set" cannot do this.
Where This Sits
Three lines of text-model work motivate the design. PromptBench and Back to Square One showed that memorized benchmark content inflates scores on MMLU and Winograd-style tasks. DyVal generated fresh reasoning problems and found that models strong on static benchmarks scored near zero on them. LiveBench refreshes its questions monthly from post-cutoff sources and verifies answers with code. All three work only on text understanding. We borrow two things: LiveBench's monthly, post-cutoff sourcing, and the rule that ground truth must be checkable without a model judge.
RealTime QA and TimeLMs supply the time lens. RealTime QA showed that stale knowledge sources collapse question-answering accuracy. TimeLMs showed that static text models lose about 10% per year on future social-media text. We carry that time lens from facts to pronunciation.
The speech-side work mostly constrains the design rather than motivating it. LibriTTS and LibriTTS-R established the benchmark whose validity we question. LibriTTS-R also showed that data quality dominates TTS quality: Non-Attentive Tacotron MOS rose from 2.4–3.4 to 4.0–4.3 after restoration. Any comparison between old and new test material must therefore hold prompt quality fixed, or quality will be mistaken for contamination. Deep Voice 3 and the emotional-TTS study of He et al. show that TTS models reproduce recording artifacts rather than discarding them. We read this as a warning: noisy fresh prompts would unfairly penalize the "fresh" arm. Miipher-2 cleans audio at scale but reports high error rates from hallucinated phonemes in hard settings. That rules out naively restoring web audio to build references.
Utterance Selection for Optimizing Intelligibility of TTS Voices Trained on ASR Data supplies a method and a caveat. Its recognizer-based intelligibility correlated with human ratings at r = 0.728–0.876, but that validation was done on in-domain text. Whether the correlation survives on novel vocabulary is one of our checks. VocBench and AHELM are the closest evaluation efforts. VocBench uses static vocoder splits. AHELM standardizes audio-understanding evaluation but does not evaluate synthesis or refresh its items. Neither addresses contamination or time.
Research Program
The study has three components.
Controlled contamination. We train our own multi-speaker TTS models on LibriTTS-R. We use two architectures that fail differently: an attention-free duration-based model in the Non-Attentive Tacotron family, and a diffusion decoder, which generates audio by gradually denoising random noise. For each architecture we inject held-out test material at controlled doses. The text arm adds test transcripts spoken by other training speakers, at zero, one or five copies. The voice arm adds a subset of test speakers' non-test utterances. A cross arm does both. Everything else stays identical, including training steps. Because we control the training data, differences between a clean model and an injected model measure contamination causally rather than guessing at it from public models whose data is unknown.
Dated evaluation slices. Each month we collect sentences from news and encyclopedia edits published after the training data's cutoff and normalize them with a fixed text front end. Every word is tagged as in-vocabulary or novel relative to the training transcripts. Novel words cover new person, place and product names and coinages. Sentences are matched to LibriTTS test sentences on length, and on phoneme and punctuation statistics. Any remaining difference then comes from novelty rather than difficulty.
Speaker prompts come from volunteers reading in a controlled room at 24 kHz, the same rate as LibriTTS. These are genuinely unseen voices. They are matched on DNSMOS, an automatic predictor of perceived audio quality, to the LibriTTS-R test prompts. A second, restored arm takes in-the-wild prompts, cleans them with a Miipher-style restorer, and compares them against clean recordings of the same volunteers. This measures how much restoration shifts the scores.
Prior-free scoring. Intelligibility is scored three ways, which lets us separate model error from evaluator error. The first is a standard large-vocabulary recognizer. The second is the same recognizer with its language prior weakened, meaning it relies more on the sound and less on which words usually follow each other. The third is paid human transcription on a stratified subset. Novel-word accuracy is also judged by listeners against a reference pronunciation read by the volunteers. Speaker fidelity is scored with speaker-encoder cosine similarity (SECS), and naturalness with MOS on a subset.
The main quantity reported is the gap between each model's static-benchmark score and its dated-slice score. That gap is decomposed into the part eliminated by removing injected contamination, the part that tracks the novel-word fraction, and the part that disappears when a prior-free evaluator replaces the standard recognizer.
Baselines
The naive reference point is the standard protocol. Each model is scored on the LibriTTS test splits with a standard recognizer, SECS and MOS, exactly as the field reports it. LibriTTS-R gives the anchor: a Non-Attentive Tacotron trained on LibriTTS-R scores MOS 4.0–4.3. No reported result was found on Lacuna for the same model on post-cutoff text or unseen speakers, and filling that hole is the point of the study. All models see identical data and step budgets. The static and dated conditions share the same synthesis settings. The protocol "beats" this baseline if it reveals a gap larger than the variation between test-set reshuffles, and if the gap re-ranks models rather than merely shifting them all down.
The strongest existing competitor is a newer, larger static benchmark: a freshly released speaker-disjoint corpus such as WenetSpeech4TTS, or a held-out LibriTTS-R split with overlap filtered by n-gram matching against the training transcripts. This is the obvious alternative a careful group would adopt today. It is also a real threat, because it may catch text contamination on its own. We run it with the same models and the same evaluators. The dated protocol must expose a component this baseline misses, either lexical novelty or evaluator repair. Otherwise the simpler baseline wins.
Two ablations isolate the key ingredients. Removing the novel-word tagging, so that fresh sentences are scored as a single pool, tests whether time-stamping adds anything beyond freshness. Replacing the prior-free evaluator with the standard recognizer tests how much evaluator contamination hides model errors.
Why not just…?
Why not just filter n-gram overlap between the test transcripts and the training data? For models trained on our own LibriTTS-R subsets this is feasible, and it is a baseline above. It cannot detect voice memorization, which is acoustic rather than textual. It cannot see the evaluator's own contamination. And it produces no post-cutoff vocabulary, since a filtered LibriTTS sentence is still Victorian prose. The injection experiments tell us directly whether text-overlap filtering captures most of the inflation. If the text arm explains nearly all of the gap, we will say so and recommend the cheaper fix.
Why not just release a new static test set? A static set is contaminated the moment it is posted, which is the lesson behind LiveBench's monthly cadence. A static set also freezes its vocabulary, so it measures lexical obsolescence only once. The comparison with WenetSpeech4TTS-style splits settles whether this matters in practice over the study's time window.
Why not just clean web audio with Miipher-2 and use it as fresh references? Miipher-2 reports high error rates caused by hallucinated phonemes in hard settings, and LibriTTS-R notes glitched outputs. A restorer could quietly rewrite exactly the rare names we want to test. The restored-prompt arm, compared against clean recordings of the same volunteers, measures this directly. If the shift is negligible, restoration becomes a cheap way to scale the protocol, and that would be a useful positive finding.
Why not just have a multimodal model judge the synthesized speech? LiveBench found that model judges fail on items beyond their own ability. The judge would share the recognizer's blind spot for unseen names, since it cannot hear a word it does not know. AHELM reports 83.3% weighted kappa with humans on captioning, a task that does not probe pronunciation. We therefore use human transcription as ground truth. On one subset we test the judge against it; it is a baseline, not the scorer.
Decisive Checks
The first check is causal inflation. At dose five, text injection should lower recognizer word error rate (WER) on the injected sentences relative to the clean model. Voice injection should raise SECS on the injected speakers. We will report whether the text effect is small next to the lexical-novelty effect, as predicted. If verbatim memorization dominates instead, the core claim fails.
The second check is lexical obsolescence. On dated slices, error should concentrate on novel words and rise with the novel-word fraction. The check is whether this holds after matching sentences on length and phoneme statistics and after human transcription.
The third check is evaluator repair. The prediction is that the gap between the standard and prior-weakened recognizers is larger on LibriTTS text than on dated text. We will also report the correlation between recognizer-based and human intelligibility on dated slices and compare it with the r = 0.728–0.876 reported in Utterance Selection. A clear drop would mean the field's standard intelligibility metric is least reliable exactly where it matters most.
The fourth check is ranking stability. Model rankings under the static and dated protocols should differ beyond bootstrap noise. If the rankings agree, the protocol measures a real but uniform offset, and the static benchmark remains adequate for comparing models even if it is inadequate for absolute claims.
References
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark
- RealTime QA: What's the Answer Right Now?
- Mitigating Temporal Misalignment by Discarding Outdated Facts
- PromptBench: A Unified Library for Evaluation of Large Language Models
- Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- TimeLMs: Diachronic Language Models from Twitter
- LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech
- LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpus
- Deep Voice 3: 2000-Speaker Neural Text-to-Speech
- Improve emotional speech synthesis quality by learning explicit and implicit representations with semi-supervised training
- Miipher-2: A Universal Speech Restoration Model for Million-Hour Scale Data Restoration
- Utterance Selection for Optimizing Intelligibility of TTS Voices Trained on ASR Data
- Vocbench: A Neural Vocoder Benchmark for Speech Synthesis
- AHELM: A Holistic Evaluation of Audio-Language Models
- WenetSpeech4TTS: A 12,800-hour Mandarin TTS Corpus for Large Speech Generation Model Benchmark