Roles of the average voice in speaker-adaptive HMM-based speech synthesis

Junichi Yamagishi, Oliver Watts, Simon King, Bela UsabaevInterspeech 2010

TL;DR

This paper investigates why some synthetic voices sound worse than others, even with the same amount of training data. They found that voices whose acoustic characteristics are further from the "average voice" tend to be less natural. This suggests that improving average voice models and adapting to them more effectively could lead to better synthetic speech quality.

What Problem Does This Paper Address?

Developing high-quality, personalized synthetic voices (Text-to-Speech, TTS) has become more accessible with HMM-based speech synthesis. This approach allows adapting a general "average voice" model to a specific target speaker using only a small amount of their speech data. However, researchers observed that even with identical adaptation data, the quality of synthetic speech can vary significantly across different target speakers. This paper aims to understand the factors contributing to this variation, particularly why a few speakers consistently produce lower-quality synthetic speech.

Methods

The authors utilize a speaker-adaptive HMM-based speech synthesis system. The core components include:

  • Speech Analysis: Extracting acoustic features (mel-cepstrum, log F0, aperiodicity) using the STRAIGHT vocoder.
  • Average Voice Training: Building context-dependent, multi-stream, tied-state HMMs from a large corpus of speech data to represent an "average" speaker. Gender-specific average voices were trained.
  • Speaker Adaptation: Transforming the average voice HMMs to a target speaker's voice using constrained maximum a posteriori (SMAP) linear regression.
  • Speech Generation: Synthesizing speech from the adapted HMMs using a parameter generation algorithm.

To analyze the quality variations, they employed:

  • Multidimensional Scaling (MDS): Visualizing the relationships between 120 synthesized voices in a low-dimensional space, based on mel-cepstral distances between their acoustic features.
  • Mean Opinion Scores (MOS): Conducting listening tests with human evaluators to rate the naturalness of the synthetic speech.

Datasets

  • WSJO Corpus: Used for training gender-dependent average voice models. This corpus provided a substantial amount of speech data (over 21 hours for males, over 24 hours for females).
  • Blizzard Challenge 2008 Test Sentences: Used for generating synthetic speech samples for evaluation.
  • 40 Listeners: Participated in the MOS listening tests.

Key Insights and Ideas

  • The "Average" is Key: The study strongly suggests that the acoustic distance between a target speaker's voice and the average voice model is a significant factor influencing synthetic speech quality. Voices closer to the average tend to sound more natural.
  • Recording Conditions Matter, But Not How We Thought: While inconsistent recording conditions can introduce acoustic differences, the paper found that these differences did not directly correlate with MOS scores. The location of recording sites (MIT, SRI, TI) did not predict quality variations.
  • "Average" Might Be More Attractive: The findings echo research in facial attractiveness, where averaged faces are often perceived as more attractive. This raises an intriguing hypothesis: perhaps the statistical averaging effect, often seen as a weakness in HMM-based synthesis, could actually contribute to more attractive-sounding voices.
  • Average Voice is Highly Natural: Surprisingly, the average voice itself received the highest naturalness scores in the listening tests, suggesting it represents a highly desirable acoustic target.

Key Results

  • A modest but significant negative correlation was found between the mel-cepstral distance from the average voice and the MOS naturalness score (correlation coefficient around -0.48). This means as the distance increases, naturalness generally decreases.
  • The paper found that 23% of the variation in synthetic speech quality could be explained by the mel-cepstral distance from the average voice. The remaining 77% is attributed to unknown factors.
  • The average voice model itself achieved a high mean MOS score (3.9).
  • No significant correlation was found between MOS scores and recording sites, indicating that acoustic differences due to recording conditions did not directly impact perceived naturalness in this study.

Limitations and Future Work

  • Unexplained Variation: The paper acknowledges that mel-cepstral distance only explains a portion (23%) of the variation in synthetic speech quality. Future work should investigate the remaining 77% of factors.
  • Attractiveness vs. Naturalness: While the findings align with attractiveness research, the authors cannot definitively conclude whether listeners were judging naturalness or attractiveness. Further investigation into this distinction is suggested.
  • Improving Average Voice Models: The findings provide a basis for developing improved training and adaptation strategies for average voice models. This could involve selecting speakers for the average model more carefully or developing adaptation techniques that better account for the distance to the average.
  • Multiple Average Voices: The study suggests that using multiple gender-dependent average voice models and selecting the closest one for adaptation might improve performance.

Summary updated

BibTeX
@inproceedings{yamagishi2010roles,
  author = {Junichi Yamagishi and Oliver Watts and Simon King and Bela Usabaev},
  title = {{Roles of the average voice in speaker-adaptive HMM-based speech synthesis}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure