Data Quality and Acoustic Fidelity in Text to Speech

146 concepts111 papers1 proposal

In automatic speech recognition (ASR), the prevailing wisdom has long been that quantity beats quality. Training a model on thousands of hours of noisy, diverse, "in-the-wild" audio forces it to learn representations that are robust to background hums, overlapping speakers, and poor microphone quality.

However, when we flip the objective from understanding speech to generating it via Text-to-Speech (TTS) synthesis, this wisdom becomes a trap. In generative speech modeling, scaling up dataset size by scraping uncurated web audio often degrades performance rather than improving it. To understand why this happens, we must look at how neural synthesis models interact with the flaws in their training data.

The Mimicry Trap and Acoustic Leakage

The fundamental reason why TTS models are sensitive to data quality lies in their objective function. Unlike ASR models, which map complex acoustic signals down to abstract text tokens, TTS models map clean text tokens up to highly detailed acoustic spaces (such as mel-spectrograms or raw waveforms). Modern neural networks are exceptionally good at capturing fine-grained statistical patterns. If the training data contains background noise, room reverberation, or microphone distortion, the model does not perceive these as corruptions to be ignored; it treats them as intrinsic characteristics of human speech to be faithfully replicated.

As Wei Ping et al. demonstrated with Deep Voice 3: 2000-Speaker Neural Text-to-Speech, scaling a system to thousands of speakers using datasets originally designed for speech recognition introduces a severe trade-off. The resulting synthetic speech frequently suffers from reduced output quality due to environmental noise and lower audio fidelity embedded in the source data.

This phenomenon, often called acoustic leakage, is particularly problematic in nuanced synthesis tasks. For instance, Jiaxu He et al. found that in emotional speech synthesis, low-quality acoustic features from the training corpora leak directly into the generated output (Improve emotional speech synthesis quality by learning explicit and implicit representations with semi-supervised training). Even when employing advanced semi-supervised strategies designed to isolate emotional expression from speaker identity, the model's output remains fundamentally constrained by the recording fidelity of the source material.

The Resolution Bottleneck

A second critical factor is the physical resolution of the audio, measured by its sampling rate. ASR datasets typically standardize on a 16kHz sampling rate because the human vocal tract's core phonetic information fits comfortably within an 8kHz Nyquist frequency.

For synthesis, however, 16kHz audio sounds muffled, robotic, and lacks the high-frequency detail required for a natural-sounding voice. High-fidelity neural vocoders require a higher frequency threshold to reconstruct clean spectral envelopes.

ASR Standard (16 kHz Sampling)  --->  [ 8 kHz Bandwidth ]  ---> Sufficient for Phoneme Recognition
TTS Standard (24+ kHz Sampling) --->  [ 12+ kHz Bandwidth ] ---> Critical for Natural Vocoder Synthesis

To bridge this gap, Heiga Zen et al. developed LibriTTS, a corpus explicitly derived from LibriSpeech but reprocessed at a 24kHz sampling rate with preserved punctuation and sentence-level segmentation (LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech). The authors showed that this increase in sampling rate, combined with cleaner data boundaries, drastically improves the subjective naturalness of synthesized speech. When building corpora for under-resourced languages, researchers like Alexander Gutkin et al. have pushed this standard even further to 48kHz to ensure that neural models can capture the full objective quality of the target language (Developing an Open-Source Corpus of Yoruba Speech).

Without this high-frequency data, models trained on low-sampling-rate historical or crowdsourced data suffer from degraded spectral envelopes (A mel-cepstral analysis technique restoring high frequency components from low-sampling-rate speech). The model simply cannot synthesize details it has never been exposed to during training.

The Alignment Breakdown

Beyond acoustic fidelity, TTS models require an absolute, deterministic alignment between text tokens and acoustic frames. If an ASR transcript misses a comma, a capitalized letter, or a brief stutter, the recognition model is largely unaffected. For a TTS model, these omissions break the attention mechanism.

Traditional text-to-speech architectures rely on pristine, studio-quality audio and perfectly aligned transcripts to function effectively (VoiceLoop: Voice Fitting and Synthesis via a Phonological Loop). When a dataset contains transcription errors or imprecise phonetic segmentations, the model's internal attention alignment matrix becomes noisy. Jindřich Matoušek et al. demonstrated that even subtle annotation errors at the word or utterance level propagate through the synthesis pipeline, leading to dropped words, unnatural pauses, and a severe degradation of overall intelligibility (Annotation errors detection in TTS corpora).

Furthermore, raw internet audio often contains overlapping speakers, background music, or abrupt cuts. As Linhan Ma et al. noted, these uncurated imperfections make raw data fundamentally unsuitable for direct TTS training, as the model cannot resolve the ambiguous mapping between a single line of text and a chaotic, multi-speaker acoustic signal (WenetSpeech4TTS: A 12, 800-hour Mandarin TTS Corpus for Large Speech Generation Model Benchmark).

Curating the Path Forward

To leverage the vast scale of the internet without destroying synthesis quality, the research community has shifted from blind data accumulation to intelligent data curation and restoration. This strategy operates on three fronts: filtering, automated preprocessing, and robust architectural design.

Intelligent Filtering

Instead of training on everything, models achieve better results by selecting only the cleanest subsets of data. Jinchuan Tian et al. demonstrated that by filtering out low-quality audio-text pairs and standardizing text formatting, speech foundation models could achieve superior performance while using 15% less training data (On the Effects of Heterogeneous Data Sources on Speech-to-Text Foundation Models). Similarly, Erica Cooper et al. proved that carefully selecting specific utterances from noisy ASR datasets significantly optimizes the intelligibility of the resulting TTS voices compared to training on the full, uncurated corpus (Utterance Selection for Optimizing Intelligibility of TTS Voices Trained on ASR Data).

Automated Preprocessing and Restoration

To salvage massive, million-hour scale datasets, researchers use specialized speech restoration models as a front-end filter. Frameworks like AutoPrep integrate state-of-the-art denoising, speaker diarization, and alignment models to transform messy podcasts and videos into structured, studio-rivaling corpora (AutoPrep: An Automatic Preprocessing Framework for In-the-Wild Speech Data). On a larger scale, universal speech restoration models like Miipher-2 leverage self-supervised speech representations to strip out environmental noise, reverberation, and compression artifacts across hundreds of languages, ensuring that generative models receive a pristine training signal (Miipher-2: A Universal Speech Restoration Model for Million-Hour Scale Data Restoration).

Robust Modeling

When noisy data cannot be avoided, architectures must be modified to handle it. For example, Jiheum Yeom et al. introduced an autoguidance mechanism in VoiceGuider: Enhancing Out-of-Domain Performance in Parameter-Efficient Speaker-Adaptive Text-to-Speech via Autoguidance. By using a weakened version of the model to steer the main synthesis network away from errors, they prevent the artifacts and poor recording conditions of "in-the-wild" reference voices from corrupting the final synthesized output.

Ultimately, the evolution of neural text-to-speech highlights a critical paradigm: while comprehension models thrive on chaos, generation models demand clarity. High-fidelity speech synthesis requires either a pristine recording environment or a highly sophisticated algorithmic filter to ensure that the model learns to replicate the human voice, and nothing else.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_data_quality_impact_on_tts_performance_ef4ea7db,
  title = {data quality impact on TTS performance},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/data-quality-impact-on-tts-performance/rdr_ef4ea7db510949639f1a5a09b17d8612},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_data_quality_impact_on_tts_performance_ef4ea7db_v1,
  title = {data quality impact on TTS performance},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/data-quality-impact-on-tts-performance/rdr_ef4ea7db510949639f1a5a09b17d8612/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake