Training Multi-Speaker Neural Text-to-Speech Systems Using Speaker-Imbalanced Speech Corpora

Hieu-Thi Luong, Xin Wang, Junichi Yamagishi, Nobuyuki NishizawaInterspeech 2019

TL;DR

This paper tackles the challenge of training high-quality multi-speaker Text-to-Speech (TTS) systems when the available data for each speaker is highly unbalanced. They propose an ensemble approach that combines multiple speaker-specific models trained on sampled data, demonstrating significant improvements in speech quality, especially for underrepresented speakers, compared to conventional single-speaker or pooled multi-speaker models.

What Problem Does This Paper Address?

Training a high-quality Text-to-Speech (TTS) system for a specific speaker typically requires a large amount of clean, well-recorded speech data. When this data is scarce, a common solution is to train a multi-speaker TTS model that can generate speech for multiple voices using a shared architecture. However, a significant challenge arises when the available speech corpora are "speaker-imbalanced," meaning some speakers have vastly more data than others. In such scenarios, simply pooling all available data can lead to models that are over-trained on majority speakers and under-trained on minority ones, resulting in degraded speech quality for the latter. This paper investigates the most effective strategies for leveraging such imbalanced corpora to achieve good performance across all speakers.

Methods

The authors explore two main strategies for handling speaker-imbalanced corpora in multi-speaker TTS:

  • Resampling Techniques: They apply under-sampling (reducing data from majority speakers) and over-sampling (increasing data from minority speakers) to create synthetically balanced training sets.
  • Ensemble Modeling: They propose a non-parametric ensemble approach. This involves training multiple "subsystems" (each being a multi-speaker TTS model with the same architecture but trained on different randomly sampled subsets of the data). The final acoustic features (Mel-generalized cepstral coefficients - MGCs, and fundamental frequency - F0) are then generated by averaging the outputs of these subsystems. This ensemble strategy is inspired by similar techniques used in speech recognition.

The core multi-speaker acoustic model is an auto-regressive neural network where a speaker-specific one-hot vector is appended to the linguistic input. This allows the network to learn shared parameters across all speakers, with only speaker-specific biases in the first hidden layer.

Datasets

The experiments were conducted using a corpus of utterances from ten female Japanese speakers. The number of unique utterances per speaker varied significantly, ranging from approximately 1,000 to 10,000. The paper provides a detailed breakdown of the training, validation, and testing sets, as well as the number of unique utterances after various sampling strategies in Table 1. No new benchmark datasets were introduced; the focus was on evaluating strategies on an existing, imbalanced corpus.

Key Insights and Ideas

  • Pooling is a reasonable baseline: Simply combining all available data from all speakers (the "MU" strategy) yields a well-rounded performance and is a good starting point, often outperforming speaker-dependent models.
  • Ensemble power: The proposed ensemble method, which combines predictions from multiple models trained on different data subsets, significantly outperforms both single-speaker and pooled multi-speaker models, particularly for underrepresented speakers. This highlights the benefit of diversity in training data and model outputs.
  • Resampling nuances: While resampling techniques (like over-sampling) can improve overall performance, they can sometimes lead to degradation for majority speakers, suggesting a trade-off. The ensemble approach seems to mitigate some of these issues.
  • Ensemble benefits outweigh costs: Despite the increased complexity and inference time, the ensemble strategy's superior performance, especially for minority speakers, makes it a compelling approach.

Key Results

  • Objective Evaluation: Figures 2 and 3 show objective metrics (mel-cepstral distortion and F0 correlation). The ensemble strategy (EN) generally achieved the best results, often outperforming the conventional multi-speaker model (MU) and speaker-dependent models (SD). Even though the ensemble model used fewer unique utterances from majority speakers due to random sampling, it still achieved competitive or superior results.
  • Subjective Evaluation: An AB preference test revealed that the ensemble model (EN) was significantly preferred over the speaker-dependent model (SD) and the pooled multi-speaker model (MU) for most speakers. The ensemble model showed substantial improvements, especially for speakers with less data. The anchor test comparing the ensemble model to a "copy synthesis" (reference) showed that while the copy synthesis was preferred, the ensemble model achieved a high level of quality.

Limitations and Future Work

  • Increased Complexity: The primary limitation of the ensemble approach is the increased number of parameters and consequently, higher training and inference times.
  • Future Directions: The authors plan to explore knowledge distillation to train a single "student" model that inherits the performance of the ensemble "teacher" model, thereby reducing complexity. They also aim to introduce diversity in network structures, in addition to data diversity, to further capitalize on the strengths of different models and mitigate weaknesses.

Summary updated

BibTeX
@inproceedings{luong2019training,
  author = {Hieu-Thi Luong and Xin Wang and Junichi Yamagishi and Nobuyuki Nishizawa},
  title = {{Training Multi-Speaker Neural Text-to-Speech Systems Using Speaker-Imbalanced Speech Corpora}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure