The IMU speech synthesis entry for Blizzard Challenge 2019

Rui Liu, Jingdong Li, Feilong Bao, Guanglai GaoBlizzard Challenge 2019 · OpenReview

TL;DR

The IMU team developed an end-to-end Mandarin text-to-speech (TTS) system for the Blizzard Challenge 2019, leveraging the Tacotron model and Griffin-Lim vocoder. While aiming for naturalness by incorporating prosodic and interjection prediction, the system underperformed, highlighting challenges in data quality, model simplicity, and the limitations of concatenating synthesized segments.

What Problem Does This Paper Address?

The paper addresses the challenge of building a high-quality Mandarin text-to-speech (TTS) voice from a given audio corpus for the Blizzard Challenge 2019. The core task is to synthesize speech that is not only intelligible but also natural-sounding and closely mimics the target speaker's style. A significant hurdle is the non-ideal quality of the provided audio data, which was not recorded in a professional studio and contained various speech phenomena like interjections and prosodic breaks that are crucial for naturalness. The challenge also involves handling diverse input text, including numbers, English words, and abbreviations, which require careful normalization for accurate synthesis.

Methods

The IMU team employed an end-to-end TTS approach, primarily using the Tacotron model for acoustic parameter generation. Their novel contribution lies in enhancing the standard Tacotron architecture with explicit prosodic and interjection prediction modules.

  • Tacotron Model: This forms the backbone of the system, taking character-level features and generating speech spectrograms.
  • Prosodic Phrase Break Prediction: A BiLSTM-based model was developed to predict prosodic break labels for each word. This aims to improve naturalness by segmenting speech into meaningful units.
  • Interjection Prediction Model: Another BiLSTM-based model was used to predict interjection labels for words. This was intended to capture emotional nuances present in the training data.
  • Feature Integration: The predicted prosodic and interjection embeddings were concatenated with character embeddings and upsampled to create a richer input representation for the Tacotron encoder.
  • Vocoder: The Griffin-Lim algorithm was used to convert the generated spectrograms into audio waveforms.
  • Synthesis Pipeline: Long sentences were first segmented, then enriched with predicted prosodic and interjection information, synthesized as short segments, and finally concatenated.

Datasets

The primary dataset used was the Mandarin talkshow corpus provided for the Blizzard Challenge 2019. This corpus contained approximately 480 utterances with a total duration of 8 hours, sampled at 24kHz and provided in MP3 format. The training data was characterized by rich emotional content, interjections, and speaker-specific prosodic breaks. The testing transcriptions included a variety of text types, such as news, facts, English words, abbreviations, numbers, letters, and Chinese poetry. No external training data was used.

Key Insights and Ideas

  • The Importance of Prosody and Emotion: The team recognized that capturing prosodic breaks and interjections is vital for naturalness and emotional expression in speech synthesis, especially for talkshow-style audio.
  • End-to-End with Enhancements: While end-to-end models like Tacotron simplify the pipeline, adding explicit modules for prosodic and emotional cues can potentially improve performance, even if the implementation has limitations.
  • Data Quality is Paramount: The non-ideal quality of the training data, including studio limitations and potential mismatches between speech and text, posed significant challenges that were addressed through preprocessing but likely impacted final quality.
  • Concatenation Artifacts: The strategy of segmenting long sentences, synthesizing them individually, and then concatenating them introduced discontinuities, negatively affecting speech quality. This highlights the difficulty of seamless long-form synthesis.

Key Results

The listening test results for the IMU system were unsatisfactory. The system ranked 19th out of all submitted systems (excluding natural speech) for naturalness ratings. It outperformed the Merlin benchmark (System B) in naturalness but lagged significantly behind the champion system (M). For speaker similarity, the system performed comparably to some participants but was generally lower than top systems, with reasons attributed to incomplete interjection prediction, lack of detailed prosodic hierarchy modeling, failure to extract speaker characteristics, and concatenation issues. The intelligibility test (pinyin error rate) showed the system at approximately 18%, ranking 12th among submitted systems.

Limitations and Future Work

The authors acknowledge several limitations:

  • Simplified Models: The interjection and prosodic prediction models are considered too simple and require further improvement.
  • Text Normalization Gaps: The text normalization process, particularly for digits, English words, letters, and ancient poetry, was insufficient.
  • Lack of Acoustic Feature Extraction: The system did not explicitly consider the speaker's acoustic characteristics for style transfer, which is a "fatal shortcoming."
  • Concatenation Issues: The method of concatenating short synthesized segments introduced discontinuities.
  • Vocoder Choice: The use of the Griffin-Lim algorithm, a traditional vocoder, is a significant limitation compared to modern neural vocoders like WaveNet, which are known to produce higher quality waveforms.

Future work could involve improving the prediction models, enhancing text normalization, incorporating speaker-specific acoustic features, developing better methods for long-sentence synthesis, and adopting neural vocoders for waveform generation.

Summary updated

Abstract

This paper decribes the IMU speech synthesis entry for Blizzard Challenge 2019, where the task was to build a voice from Mandarin audio data. Our system is a typical end-to-end speech synthesis system. The acoustic parameters is modeled by using “Tacotron” model, and the vocoder is using Griffin-Lim algorithm. In the synthesis stage, the task is divided into the following parts: 1) segment long sentence into short sentences by comma; 2) predict interjection labels of each words in short sentences; 3) predict prosodic break labels of each words in short sentences; 4) generate corresponding synthesis speech for each short sentences which enriched by prosodic break labels and interjections; 5) concatenate short sentences into an entire long sentence.
BibTeX
@inproceedings{liu2019imu,
  author = {Rui Liu and Jingdong Li and Feilong Bao and Guanglai Gao},
  title = {{The IMU speech synthesis entry for Blizzard Challenge 2019}},
  booktitle = {Blizzard Challenge},
  year = {2019},
  url = {https://openreview.net/forum?id=8DQurLp3E9},
}
Figure