Submission from vivo for Blizzard Challenge 2019

Yang Wang, Mingyue Wu, Zhiming Xu, Baoqin Luo, Hao Liang, Binbin Chen, Xiaoxin ChenBlizzard Challenge 2019 · OpenReview

TL;DR

This paper presents vivo's submission to the Blizzard Challenge 2019, focusing on building an expressive Chinese speech synthesis system using an end-to-end Tacotron-based model. They introduce several practical improvements, including better speech energy normalization, outlier removal for cleaner training data, and explicit modeling of non-speech sounds like breaths and clicks, leading to a system that achieved moderate results in naturalness and similarity evaluations.

What Problem Does This Paper Address?

The Blizzard Challenge aims to push the boundaries of speech synthesis by providing a common dataset and task for researchers to evaluate and compare their systems. This year's challenge focused on creating an expressive speech synthesis system using an 8-hour corpus of a Chinese talk-show character. Building high-quality, natural-sounding, and expressive speech synthesis systems, especially for languages with complex character-to-sound mappings like Chinese, remains a significant challenge. Traditional methods often require extensive manual tuning and domain expertise, while end-to-end deep learning models, though promising, can still struggle with nuances like prosody, non-speech sounds, and data imperfections. This paper tackles these challenges by adapting and improving an existing end-to-end Tacotron model for this specific task.

Methods

The core of vivo's system is the Tacotron architecture, a popular end-to-end sequence-to-sequence model for text-to-speech synthesis. They built upon an open-source implementation (Mozilla TTS Tacotron) and introduced several key improvements:

  • Speech Energy Normalization: A more robust method for normalizing speech energy was implemented. The original approach had a counter-intuitive normalization range. The authors modified this to use a clear lower and upper bound, which they tuned through grid search, significantly alleviating energy abnormality issues in synthesized speech.
  • Outlier Removal: The training data, consisting of long utterances, was automatically segmented into shorter ones. This process, however, sometimes resulted in problematic segments (e.g., cutting off mid-syllable, unusual background noise, or laughter). The authors identified and removed these outliers, which improved alignment and reduced trailing silence abnormalities. Interestingly, they initially removed poems as outliers due to their distinct prosody but had to re-introduce them for the evaluation set.
  • Special Phone Modelling: To capture the expressive and spontaneous nature of the talk-show corpus, which included breath sounds and mouth clicks, the authors introduced special "phone" models for these non-speech events. This explicit modeling aimed to prevent these sounds from obscuring common speech sounds and improve prosody and intelligibility. They also attempted to model "gasp" but merged it with audible breath due to annotation difficulty.

Datasets

The primary dataset used was the 8-hour corpus provided for the Blizzard Challenge 2019, featuring a well-known Chinese talk-show character. No external corpora were used for training. The text transcriptions were found to be imperfect, requiring manual correction.

Key Insights and Ideas

  • Practical Improvements Matter: While end-to-end models are powerful, small, well-motivated engineering improvements (like better normalization and data cleaning) can have a significant impact on speech quality.
  • Data Quality is Crucial for Expressiveness: The spontaneous nature of the talk-show corpus, with its non-speech sounds, highlighted the need for explicit modeling of these elements to achieve natural expressiveness. Simply ignoring them or letting them be implicitly learned can lead to degraded speech.
  • Outliers Can Be Tricky: Identifying and handling outliers in speech data is not always straightforward. What might seem like an outlier (e.g., poetry) can be a significant part of the evaluation data, requiring careful consideration and potentially re-training.
  • Segmentation Trade-offs: Automatically segmenting long utterances is necessary for many models, but it introduces its own set of problems that need to be addressed through outlier removal and careful handling of segment boundaries.

Key Results

The evaluation results, presented as boxplots comparing various systems, showed that vivo's system achieved middle-level scores for both naturalness and similarity. While not among the top-performing systems, the authors considered their trials "somewhat successful," indicating that their improvements contributed positively to the overall system performance. Specifically, the energy normalization addressed a noticeable abnormality, outlier removal improved alignment and reduced trailing silence issues, and special phone modeling notably improved speech prosody and intelligibility.

Limitations and Future Work

The authors acknowledge two main limitations:

  • Stop-net Malfunction: A small number of synthesized long sentences terminated abruptly, suggesting that the stop-net module (which determines when synthesis should end) did not always function correctly. This module needs improvement.
  • Concatenation Artifacts: The system segmented text into shorter sentences for synthesis and then concatenated them. This "divide-and-conquer" approach led to noticeable speech background switching and a lack of expressive consistency across consecutive sentences within a paragraph.

Future work could explore synthesis mechanisms capable of handling entire paragraphs directly to overcome these concatenation artifacts.

Summary updated

Abstract

This paper presents the vivo speech synthesis system for Blizzard Challenge 2019. The task is to build an expressive speech synthesis system on an 8-hour corpus of a well-known Chinese talk-show character. Our system is based on Tacotron with several minor improvements, which are more clear speech energy normalization, outlier removal of problematic shorter utterances, and special phone modelling for explicit long silences, audible breath sounds, and mouth-click sounds. Evaluation results showed that the proposed system is somewhat successful.
BibTeX
@inproceedings{wang2019submission,
  author = {Yang Wang and Mingyue Wu and Zhiming Xu and Baoqin Luo and Hao Liang and Binbin Chen and Xiaoxin Chen},
  title = {{Submission from vivo for Blizzard Challenge 2019}},
  booktitle = {Blizzard Challenge},
  year = {2019},
  url = {https://openreview.net/forum?id=kT5xPU793Y},
}
Figure