TL;DR
This paper presents vivo's submission to the Blizzard Challenge 2019, focusing on building an expressive Chinese speech synthesis system using an end-to-end Tacotron-based model. They introduce several practical improvements, including better speech energy normalization, outlier removal for cleaner training data, and explicit modeling of non-speech sounds like breaths and clicks, leading to a system that achieved moderate results in naturalness and similarity evaluations.
What Problem Does This Paper Address?
The Blizzard Challenge aims to push the boundaries of speech synthesis by providing a common dataset and task for researchers to evaluate and compare their systems. This year's challenge focused on creating an expressive speech synthesis system using an 8-hour corpus of a Chinese talk-show character. Building high-quality, natural-sounding, and expressive speech synthesis systems, especially for languages with complex character-to-sound mappings like Chinese, remains a significant challenge. Traditional methods often require extensive manual tuning and domain expertise, while end-to-end deep learning models, though promising, can still struggle with nuances like prosody, non-speech sounds, and data imperfections. This paper tackles these challenges by adapting and improving an existing end-to-end Tacotron model for this specific task.
Methods
The core of vivo's system is the Tacotron architecture, a popular end-to-end sequence-to-sequence model for text-to-speech synthesis. They built upon an open-source implementation (Mozilla TTS Tacotron) and introduced several key improvements:
- Speech Energy Normalization: A more robust method for normalizing speech energy was implemented. The original approach had a counter-intuitive normalization range. The authors modified this to use a clear lower and upper bound, which they tuned through grid search, significantly alleviating energy abnormality issues in synthesized speech.
- Outlier Removal: The training data, consisting of long utterances, was automatically segmented into shorter ones. This process, however, sometimes resulted in problematic segments (e.g., cutting off mid-syllable, unusual background noise, or laughter). The authors identified and removed these outliers, which improved alignment and reduced trailing silence abnormalities. Interestingly, they initially removed poems as outliers due to their distinct prosody but had to re-introduce them for the evaluation set.
- Special Phone Modelling: To capture the expressive and spontaneous nature of the talk-show corpus, which included breath sounds and mouth clicks, the authors introduced special "phone" models for these non-speech events. This explicit modeling aimed to prevent these sounds from obscuring common speech sounds and improve prosody and intelligibility. They also attempted to model "gasp" but merged it with audible breath due to annotation difficulty.
Datasets
The primary dataset used was the 8-hour corpus provided for the Blizzard Challenge 2019, featuring a well-known Chinese talk-show character. No external corpora were used for training. The text transcriptions were found to be imperfect, requiring manual correction.
Key Insights and Ideas
- Practical Improvements Matter: While end-to-end models are powerful, small, well-motivated engineering improvements (like better normalization and data cleaning) can have a significant impact on speech quality.
- Data Quality is Crucial for Expressiveness: The spontaneous nature of the talk-show corpus, with its non-speech sounds, highlighted the need for explicit modeling of these elements to achieve natural expressiveness. Simply ignoring them or letting them be implicitly learned can lead to degraded speech.
- Outliers Can Be Tricky: Identifying and handling outliers in speech data is not always straightforward. What might seem like an outlier (e.g., poetry) can be a significant part of the evaluation data, requiring careful consideration and potentially re-training.
- Segmentation Trade-offs: Automatically segmenting long utterances is necessary for many models, but it introduces its own set of problems that need to be addressed through outlier removal and careful handling of segment boundaries.
Key Results
The evaluation results, presented as boxplots comparing various systems, showed that vivo's system achieved middle-level scores for both naturalness and similarity. While not among the top-performing systems, the authors considered their trials "somewhat successful," indicating that their improvements contributed positively to the overall system performance. Specifically, the energy normalization addressed a noticeable abnormality, outlier removal improved alignment and reduced trailing silence issues, and special phone modeling notably improved speech prosody and intelligibility.
Limitations and Future Work
The authors acknowledge two main limitations:
- Stop-net Malfunction: A small number of synthesized long sentences terminated abruptly, suggesting that the stop-net module (which determines when synthesis should end) did not always function correctly. This module needs improvement.
- Concatenation Artifacts: The system segmented text into shorter sentences for synthesis and then concatenated them. This "divide-and-conquer" approach led to noticeable speech background switching and a lack of expressive consistency across consecutive sentences within a paragraph.
Future work could explore synthesis mechanisms capable of handling entire paragraphs directly to overcome these concatenation artifacts.
Summary updated