TL;DR
This paper tackles the challenge of building high-quality Text-to-Speech (TTS) systems from readily available but noisy "found speech" data. They propose and evaluate various metrics to automatically identify and discard utterances that degrade TTS performance, demonstrating that a carefully selected subset of data yields better results than using all available noisy data.
What Problem Does This Paper Address?
Building a high-quality TTS system typically relies on large, clean, and phonetically balanced datasets recorded by a single speaker. However, this is often impractical, especially for low-resource languages or personalized voices for individuals with speech impairments. The authors propose leveraging "found speech" – data readily available online (e.g., YouTube, radio broadcasts, audiobooks) or through crowdsourcing. This found speech is abundant but suffers from various issues: channel variations (different microphones, recording environments), background noise, and misalignments between the audio and its transcript (e.g., coughs, laughter). While some noise can add useful diversity, too much can severely degrade TTS system performance. The core problem is how to effectively select the "good" utterances from this noisy pool to train a robust TTS system, avoiding the detrimental effects of bad data.
Methods
The paper investigates various metrics for utterance selection at the utterance level. They categorize errors into two main types: misalignment errors (audio doesn't match transcript) and channel variation errors (noise, different recording conditions).
The core methodology involves:
- Metric Evaluation: They propose and evaluate several metrics to detect these problematic utterances. These include:
- Mean Cepstral Distortion (MCD): A measure of the difference between the spectral characteristics of synthesized and true speech. Higher MCD suggests poorer synthesis quality.
- Duration: Comparing predicted vs. true utterance durations.
- Modulation Spectrum: Capturing temporal dynamics of spectral envelopes.
- Global Variance: Analyzing differences in the overall spectral variance.
- Cross-correlation based measures: Including direct wavefile cross-correlation, and cross-correlation of Teager Energy Operator and Hilbert envelope.
- Instantaneous Frequency: Measuring differences in the instantaneous frequency of signals.
- Iterative Selection: The best-performing metrics are used in an iterative process. A small "seed" set of utterances is used to train an initial model. Then, a fixed percentage (10%) of the "best" utterances (according to the metric) are added to the training set, and the model is retrained. This process is repeated to see how performance changes as more data is added.
- Two Scenarios: They evaluate two scenarios: (1) starting with a small seed set, and (2) building a model from scratch without a seed set.
- Re-alignment vs. Re-clustering: They compare two retraining strategies: re-clustering existing state models with fixed labels, and re-aligning the data and then re-clustering.
Datasets
The authors used the following datasets for training and evaluation:
- Artificially Degraded Clean Corpus: Created from the Arctic dataset (speaker RMS). This dataset was specifically engineered to provide ground truth for evaluating metrics. It includes:
- 100 utterances with shifted acoustic-transcription alignments (misalignment).
- 100 utterances convolved with impulse responses to simulate channel variations.
- A mix of both types of degradation.
- Single-Speaker Found Data: Subsets of speeches from the American President. One subset was intentionally noisy with claps and reverberation.
- Multi-Speaker Found Data:
- Male CallHome Corpus: Unscripted telephone conversations.
- Female CallHome Corpus: Similar to the male corpus.
Key Insights and Ideas
- "Not all data is good data": The central thesis is strongly supported. Using a carefully selected subset of found speech significantly outperforms using the entire noisy dataset.
- MCD is King (for detection): Mean Cepstral Distortion emerges as the most effective metric for detecting both misaligned and noisy utterances, especially when trained on a comprehensive dataset.
- Misalignment is easier to detect than channel noise: Metrics are generally better at identifying audio-transcript mismatches than subtle variations in recording conditions.
- Seed set not strictly necessary: While a seed set can be helpful, the authors found that building a model from the entire noisy dataset and then iteratively selecting utterances can yield comparable or even better results.
- Re-clustering vs. Re-alignment: For multi-speaker datasets, re-clustering with fixed labels seems more stable and predictable than re-aligning at each iteration, which can lead to erratic behavior. However, re-alignment can yield lower MCD on single-speaker datasets.
- Data selection scales: The proposed method for pruning noisy data remains effective even when a significant portion (50%) of the dataset is artificially misaligned.
Key Results
- MCD Performance: On the artificially degraded set, MCD achieved near-perfect detection accuracy (99% for misalignment, 88% for channel noise, 95% for mixed noise) when trained on a small seed set, and even higher (100%, 94%, 96.5%) when trained on the full dataset.
- Baseline Improvement: The iterative MCD selection process, even without realignment, consistently improved performance over the baseline (training on all noisy data) across various datasets, achieving lower MCD values. For example, on artificially misaligned data, the MCD was reduced by 0.3 compared to the baseline.
- Cross-correlation metrics underperform: Surprisingly, cross-correlation based measures were not good indicators of misaligned data, and global spectral measures performed poorly in detecting channel conditions.
- Duration is a strong secondary metric: The duration parameter also showed good performance in detecting misaligned data.
Limitations and Future Work
- Limitations:
- The study primarily focused on English corpora, and future work should investigate performance on non-English datasets.
- The paper explored utterance-level selection; more granular selection (phoneme, frame) could be beneficial.
- Experiments were conducted using only one metric at a time.
- Future Work:
- Test methods on large non-English corpora (e.g., Babel datasets) containing telephone speech.
- Investigate combinations of multiple metrics for more robust selection.
- Explore methods for finding source sets that best encode target sets based on pairwise dissimilarities.
- Further investigate the trade-offs between re-alignment and re-clustering strategies for different data types.
Summary updated