TL;DR
This paper investigates how to select the best audio data for training text-to-speech (TTS) systems, particularly for low-resource languages. They found that selecting data based on speaker characteristics is more effective than selecting individual utterances, leading to significantly more intelligible synthetic voices.
What Problem Does This Paper Address?
Building high-quality Text-to-Speech (TTS) systems for new languages is incredibly expensive and time-consuming, often requiring millions of dollars and extensive professional recording sessions. This is a major barrier for the vast majority of the world's languages, which lack the resources for such efforts. While previous research has explored using "found" audio data (like recordings collected for speech recognition), a key challenge remains: how to efficiently select the best subset of this often noisy and heterogeneous data to train a good TTS voice. This paper tackles this by comparing two data selection strategies: selecting based on individual utterances versus selecting based on entire speakers, aiming to improve intelligibility and reduce the cost of TTS development.
Methods
The researchers used the "Merlin" toolkit, a deep-learning-based TTS system, to train their voices. They experimented with two main data selection approaches:
- Utterance-based selection: Choosing individual audio clips based on acoustic features.
- Speaker-based selection: Choosing entire speakers whose audio exhibits certain acoustic characteristics.
They extracted a range of acoustic features (like pitch, energy, and speaking rate) from the audio data using the Praat toolkit. To evaluate the quality of the synthesized speech, they employed two methods:
- Objective evaluation: Using the IBM Watson Speech to Text API to calculate Word Error Rate (WER) on synthesized sentences.
- Subjective evaluation: Using Amazon Mechanical Turk (MTurk) to have human listeners transcribe synthesized speech and calculate WER.
They also explored Mel-Cepstral Distortion (MCD) as an objective metric that correlates well with human intelligibility.
Datasets
The primary dataset used was the Macrofone corpus, an approximately 160-hour collection of telephone speech from around 5,000 American English speakers, originally collected for Automatic Speech Recognition (ASR) systems. This dataset was chosen to simulate the conditions of training TTS on low-resource languages where professional TTS corpora are scarce.
Key Insights and Ideas
- Speaker matters more than individual utterances: The most significant finding is that selecting data based on speaker characteristics consistently leads to more intelligible TTS voices compared to selecting individual utterances, even when using the same amount of training data. This suggests that speaker consistency and vocal traits are crucial for building good TTS.
- ASR data is a viable source: The study demonstrates that readily available ASR corpora can be effectively leveraged for TTS training, opening doors for low-resource languages.
- Objective metrics can guide selection: Features like low ASR WER for a speaker, mean energy, and speaking rate proved useful for selecting good training data.
- Combining features yields better results: Jointly considering multiple acoustic features for speaker selection significantly outperformed single-feature selection, leading to the best intelligibility scores.
- MCD is a promising objective metric: Mel-Cepstral Distortion showed a strong correlation with human intelligibility evaluations, making it a valuable tool for pre-selecting or evaluating TTS voices, especially when ASR systems are not readily available.
Key Results
- The best speaker-selected voice achieved a human-evaluated WER of 29.0%, a substantial improvement over a baseline voice trained on randomly selected utterances, which had a WER of 42.4%.
- In objective evaluations using Watson ASR, speaker-selected voices generally achieved lower WERs than utterance-selected voices.
- Combining features for speaker selection led to the most intelligible voices, with the top voice achieving a WER of 29.0% on MTurk.
- The study found strong correlations (up to 91.2%) between MTurk WER and Watson WER for joint-feature-based voices, validating the use of ASR for preliminary evaluation.
Limitations and Future Work
The authors acknowledge that their work primarily focused on American English telephone speech. Future work includes extending these methods to other corpora and low-resource languages, including those from the IARPA BABEL project. They also plan to further investigate the characteristics of optimal training subsets, explore the trade-offs between utterance count and length, and develop more automatic, machine learning-based methods for data selection. The reliance on ASR systems for preliminary evaluation in low-resource languages, while practical, is noted as a potential circularity that warrants further consideration.
Summary updated