Thousands of voices for HMM-based speech synthesis

Junichi Yamagishi, Bela Usabaev, Simon King, Oliver Watts, John Dines, Jilei Tian, Rile Hu, Yong Guan, Keiichiro Oura, Keiichi Tokuda, Reima Karhila, Mikko Kurimo2009 · DOI

TL;DR

This paper explores a novel approach to generating a vast number of synthetic voices by leveraging large-scale Automatic Speech Recognition (ASR) corpora. By adapting speaker-adaptive HMM-based speech synthesis techniques, the authors demonstrate the feasibility of creating thousands of distinct voices, significantly expanding the possibilities beyond traditional, resource-intensive TTS methods.

What Problem Does This Paper Address?

Traditionally, building high-quality synthetic voices for Text-to-Speech (TTS) systems requires extensive, carefully curated speech data recorded in controlled studio environments. This data is expensive and time-consuming to collect, severely limiting the number of available voices. The authors address this bottleneck by investigating whether readily available, but less controlled, ASR corpora, which often contain recordings from numerous speakers, can be effectively utilized to create a large and diverse collection of synthetic voices.

Methods

The core of the paper lies in the application of speaker-adaptive HMM-based speech synthesis (HTS) techniques. Instead of training a separate model for each voice from scratch, the system uses an "average voice model" and then adapts it to individual speakers. This adaptation process is crucial for handling the "non-ideal" nature of ASR data, which can include variations in recording conditions, microphones, and phonetic balance. The authors leverage existing HTS frameworks, specifically mentioning the "HTS-2007/2008" system, which was designed for speaker adaptation.

Datasets

The primary datasets used are large ASR corpora:

  • Wall Street Journal (WSJ) databases: WSJ0, WSJ1, and WSJCAM0. These were particularly important for comparison and formed the basis for many of the trained voices.
  • Resource Management (RM)
  • Globalphone
  • Speecon: Finnish and Mandarin versions.
  • Japanese JNAS

For comparison, the CMU-ARCTIC speech database, a more traditional TTS corpus, was also used. The authors note that ASR corpora often contain a large number of speakers, with each speaker typically recording fewer than 100 utterances, which presents a unique challenge compared to TTS corpora where speakers might record hundreds of sentences.

Key Insights and Ideas

  • ASR Corpora as a Goldmine for TTS Voices: The paper demonstrates that ASR corpora, despite their less controlled nature, are highly suitable for building speaker-adaptive TTS systems, enabling the creation of thousands of voices.
  • Speaker Adaptation is Key: The robustness of speaker-adaptive HMMs is critical for overcoming the imperfections in ASR data, making it possible to adapt an average model to a wide range of speakers.
  • Phonetic Coverage Matters: ASR corpora, when pooled across many speakers, can offer better overall phonetic coverage than traditional TTS corpora, even if individual speakers contribute less data.
  • Geographical Representation as a Visualization Tool: To manage and showcase the vast number of generated voices, the authors developed an interactive geographical map, providing an intuitive way to explore and listen to different speakers.

Key Results

  • Thousands of Voices Created: The authors successfully generated over 1500 synthetic voices from various ASR corpora.
  • Objective Evaluation: Speaker-adaptive systems using more training data for the average voice model ("condition 2") generally outperformed those using less data ("condition 1"). Speaker-adaptive systems were found to outperform speaker-dependent systems when the target speaker's data was less than about 1 hour.
  • Subjective Evaluation: Perceptual tests mirrored the objective findings, with listeners generally preferring speaker-adaptive systems with more data. For less than 1 hour of target speaker data, adaptive systems were superior. At around 1-2 hours of target speaker data, speaker-dependent systems began to show comparable or slightly better performance in naturalness and intelligibility.
  • Geographical Demo: An interactive online demo was created, showcasing the voices and their geographical distribution.

Limitations and Future Work

  • Quality Variation: Informal listening revealed that some speakers' synthetic voices sounded worse than others, potentially due to poor phonetic coverage for that speaker or inherent voice/recording quality issues. This requires further investigation.
  • Speaker Consistency: For some speakers, inconsistencies in speaking style over long recording sessions might affect objective measures like RMSE of log Fo.
  • Future Work: The authors plan to analyze the differences in quality among the generated voices and explore applications such as medical voice banking or virtual environments. They also aim to add interactive TTS features for more languages to their geographical demonstration.

Summary updated

Abstract

TEST 02 - Elsevier's Scopus, the largest abstract and citation database of peer-reviewed literature. Search and access research from the science, technology, medicine, social sciences and arts and humanities fields.
BibTeX
@misc{yamagishi2009thousands,
  author = {Junichi Yamagishi and Bela Usabaev and Simon King and Oliver Watts and John Dines and Jilei Tian and Rile Hu and Yong Guan and Keiichiro Oura and Keiichi Tokuda and Reima Karhila and Mikko Kurimo},
  title = {{Thousands of voices for HMM-based speech synthesis}},
  year = {2009},
  doi = {10.21437/interspeech.2009-140},
}
Figure