Selection and Training Schemes for Improving TTS Voice Built on Found Data

F.-y. Kuo, I.c. Ouyang, S. Aryal, Pierre LanchantinInterspeech 2019

TL;DR

This paper explores how to build better Text-to-Speech (TTS) voices from readily available "found" audio data, which is often messy and inconsistent. They demonstrate that carefully selecting a smaller subset of this data and fine-tuning existing TTS models can significantly improve voice naturalness compared to training on the entire, unfiltered dataset.

What Problem Does This Paper Address?

Creating high-quality, natural-sounding TTS voices typically requires large amounts of carefully curated and recorded speech data. However, such data is expensive and time-consuming to collect. "Found data" – audio from sources like web videos, radio, and audiobooks – is abundant and often free, but it presents significant challenges: inconsistent recording quality, diverse speaking styles, inaccurate transcriptions, and even missing speakers (due to time, death, or illness). This heterogeneity can degrade TTS model performance. The paper addresses the problem of effectively leveraging this abundant but imperfect found data to build natural-sounding TTS voices by developing intelligent data selection and training strategies.

Methods

The core of this work lies in developing and evaluating sophisticated data selection metrics and training schemes.

  • Data Selection Metrics: The authors utilize a comprehensive set of 14 metrics to identify and reject problematic segments of found data. These metrics fall into categories like:
  • Transcription Reliability: Phone Matching Error Rate (PMER) to detect alignment errors.
  • Feature Estimation Quality: Voiced/Unvoiced Mismatch Rate for pitch estimation issues.
  • Recording Quality: Signal-to-Noise Ratio (SNR) and Mean Opinion Score Listening Quality Objective (MOS-LQO) to filter out noisy or poor-quality audio.
  • Articulation & Style Variation: Metrics like Non-fluency, Standard Deviation of Syllable Duration, Standard Deviation of F0, and Articulation score to capture variations in speaking style and pronunciation.
  • Utterance Length: To reject segments that are too short or too long.
  • Speaking Rate: Mean and Standard Deviation of Syllable Duration.
  • Selection Schemes: They define three sets of thresholds (v1, v2, v3) for these metrics, ranging from less aggressive (using "knee points" from cumulative distribution plots) to more aggressive (using "half-data points").
  • Training Schemes: Two primary training approaches are investigated:
  • Training from Scratch: Building a TTS model from the selected data without prior knowledge.
  • Fine-tuning: Taking a model pre-trained on the entire found dataset and further training it on the selected subset. This leverages the general knowledge from the full dataset while adapting to the cleaner, selected data.
  • TTS Architecture: A standard deep feed-forward neural network (DNN) architecture, commonly used in modern TTS systems, is employed. This model maps linguistic features to acoustic features (Mel-Cepstral Coefficients, band aperiodicities, fundamental frequency) using the WORLD vocoder.

Datasets

The paper uses two 24-hour single-speaker corpuses of "found data" collected from diverse sources. These datasets consist of speeches from politicians, exhibiting a wide range of recording conditions and speaking styles. After initial processing (loosely transcribing and extracting time boundaries), approximately 27 hours of data remained for each speaker. A small, held-out test set of 20 utterances (about 2.6 minutes) was used for evaluation.

Key Insights and Ideas

  • Less is More (with careful selection): The most striking finding is that a significantly smaller, carefully curated subset of found data can lead to a more natural-sounding TTS voice than training on the entire, unfiltered dataset. This challenges the common intuition that more data is always better.
  • Fine-tuning is Powerful: Fine-tuning a TTS model pre-trained on the full dataset, and then further training it on a selected subset, yields substantial improvements in naturalness. This approach seems to allow for more aggressive data selection because the model already has a baseline understanding.
  • Data Selection Trade-offs: While data selection generally improves naturalness, the paper highlights that the aggressiveness of selection can have different impacts depending on the training scheme. For fine-tuned models, more aggressive selection (half-data thresholds) is beneficial. For models trained from scratch, there's a trade-off: more aggressive selection can lead to better quality but might require more data than what's available in the aggressively selected set.
  • Metric Sets Matter, But Not Universally: The paper proposes new metric sets (v1, v2, v3) and investigates their effectiveness. While they generally lead to improvements, the authors note that further investigation is needed to find the optimal combination of metrics.

Key Results

  • Objective Evaluation: Fine-tuned systems generally outperformed systems trained from scratch across various objective metrics (MCD, F0-RMSE, DUR-RMSE, V/UV). Data selection generally led to better results for Speaker A in terms of F0-RMSE, duration, and V/UV error, but slightly worse MCD. For Speaker B, selection improved MCD but worsened F0-RMSE and V/UV error, except for one specific aggressive selection (v1 half-data) with fine-tuning, which achieved the best results.
  • Subjective Evaluation:
    • Test 1 (Selected vs. Full): All evaluated data selections led to significant improvements in naturalness compared to training on the full corpus for both speakers.
  • Test 2 (Aggressive vs. Less Aggressive Selection): For fine-tuned models, more aggressive (half-data) selections were significantly preferred over less aggressive (knee-point) selections in Speaker B. For models trained from scratch, Speaker A preferred less aggressive selections.
  • Test 3 (Fine-tuned vs. From Scratch): Fine-tuned systems were significantly more preferable than systems trained from scratch across all selection sets. This effect was more pronounced with smaller, more aggressively selected datasets.
  • Test 4 (Metric Sets): The proposed new metric sets did not significantly improve quality over existing ones, suggesting further research is needed for optimal metric combinations.

Limitations and Future Work

  • Limitations:
    • The effectiveness of the proposed metric sets (v1, v2, v3) for improving quality was not definitively established, indicating a need for further research into optimal metric combinations.
  • The paper focuses on single-speaker TTS. The applicability to multi-speaker systems is not explored.
  • The found data was limited to speeches from politicians, which might not represent the full spectrum of "found data" available.
  • Future Work:
    • Investigate optimal combinations of selection metrics through more extensive experiments.
  • Explore the benefits of these data selection strategies in more recent TTS architectures, such as sequence-to-sequence models (e.g., Char2Wav, Tacotron).
  • Potentially extend the work to multi-speaker TTS systems.

Summary updated

BibTeX
@inproceedings{kuo2019selection,
  author = {F.-Y. Kuo and I.C. Ouyang and S. Aryal and Pierre Lanchantin},
  title = {{Selection and Training Schemes for Improving TTS Voice Built on Found Data}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure