The TJU-Didi-Huiyan system for Blizzard Challenge 2019

Ju Zhang, Shaotong Guo, Cheng Gong, Shuaiting Chen, Yuguang Wang, Longbiao Wang, Wei Zou, Xiangang LiBlizzard Challenge 2019 · OpenReview

TL;DR

The TJU-Didi-Huiyan team presents their end-to-end Mandarin text-to-speech (TTS) system for the Blizzard Challenge 2019, leveraging the Tacotron 2 architecture. Their key innovation lies in a robust front-end that handles the complexities of Chinese text, including character-to-pinyin conversion with tone and prosody, to achieve natural-sounding speech.

What Problem Does This Paper Address?

Creating natural and intelligible synthetic speech, especially for tonal languages like Mandarin, is a significant challenge in TTS research. Traditional TTS systems often involve complex, multi-stage pipelines requiring expert knowledge and extensive hand-tuning. The Blizzard Challenge aims to push the boundaries of TTS by evaluating systems on metrics like naturalness, intelligibility, and similarity to a target speaker. This paper specifically addresses the difficulties of synthesizing Mandarin speech, which lacks explicit word boundaries and relies heavily on tone for meaning, by developing an end-to-end system that automates much of the complex front-end processing.

Methods

The TJU-Didi-Huiyan system employs an end-to-end approach built upon the Tacotron 2 architecture. The system is divided into two main parts:

  • Front-End Processing: This is a crucial component for Mandarin. It involves:
  • Text Regularization: Handling non-standard characters and symbols to ensure correct pronunciation.
  • Word Segmentation: Identifying word boundaries in Chinese text, which lacks explicit delimiters like spaces.
  • Grapheme-to-Phoneme (G2P) Conversion: Translating Chinese characters into Pinyin sequences, including tone information.
  • Prosody Prediction: Estimating pitch, duration, and pauses to capture the natural rhythm and intonation of speech.
  • Lexicon and Polyphone Handling: Using a pronunciation dictionary and addressing characters with multiple pronunciations. The paper highlights that while they have developed automatic methods for these tasks, manual annotation was used for some parts to reduce errors in the challenge.

  • Back-End Processing: This part utilizes the Tacotron 2 model.

  • Encoder-Decoder Architecture: Tacotron 2's encoder processes the Pinyin sequence into a hidden representation, and the decoder generates mel-spectrogram frames.
  • Attention Mechanism: A location-sensitive attention mechanism is used to align the encoder and decoder outputs.
  • Spectrogram Prediction: The system predicts linear spectrograms (rather than mel-spectrograms) because they found them more suitable for the Griffin-Lim algorithm. A "CBHG" module is used to extract features from the linear spectrogram before projection.
  • Vocoding: The Griffin-Lim algorithm is used to convert the predicted spectrograms into audible speech waveforms.

Datasets

The training data for this system consisted of approximately 8 hours of Mandarin speech from a male speaker, provided by the Blizzard Challenge committee. This data was comprised of 480 MP3 audio files, each about 60 seconds long. The paper mentions that the audio files were converted to WAV format and segmented into shorter clauses for better model learning.

Key Insights and Ideas

  • The Criticality of Front-End for Mandarin TTS: The paper strongly emphasizes that for tonal languages like Mandarin, a sophisticated front-end processing pipeline is essential. Simply converting characters to Pinyin isn't enough; capturing tone, prosody, and handling word segmentation are vital for intelligibility and naturalness.
  • Linear Spectrograms for Griffin-Lim: The choice to predict linear spectrograms instead of mel-spectrograms is a notable deviation from standard Tacotron 2 implementations. This decision was driven by empirical findings that linear spectrograms worked better with the Griffin-Lim vocoder, highlighting the interplay between different components in an end-to-end system.
  • The "CBHG" Module's Role: The inclusion of the "CBHG" module before the final spectrogram projection is presented as an important step for extracting relevant features from the linear spectrogram, contributing to better waveform generation.
  • Balancing Automation and Manual Annotation: The team's approach of developing automatic front-end processing but using manual annotation for critical parts (like prosody and word segmentation) demonstrates a practical strategy for improving performance in competitive challenges where accuracy is paramount.

Key Results

The paper presents results from the Blizzard Challenge 2019, where their system was named 'X'.

  • Naturalness: The system performed better than the Merlin benchmark on naturalness tests. However, it was still outperformed by natural speech and other state-of-the-art systems.
  • Intelligibility: The Pinyin error rate (PER) without tone was 25.9%, and the Pinyin error rate with tone (PTER) was 27.8%. The authors consider these results "reasonable," attributing them partly to the pre-processing scripts used during training.
  • Similarity: The system achieved a similarity score comparable to the Merlin benchmark. It was significantly different from several other participating systems (A, M, S, Y, Z, E, and C).

Limitations and Future Work

  • Naturalness Gap: The authors acknowledge that while their system is better than the Merlin benchmark, there's still a significant gap in naturalness compared to human speech and top-tier systems.
  • Future Directions: The team plans to investigate:
  • Code-switched speech synthesis.
  • Quantization for deep neural networks.
  • Improving performance across all evaluation criteria for future challenges.

Summary updated

Abstract

In this paper, we introduce an end-to-end text-to-speech system based on Tacotron 2 for Blizzard Challenge 2019. The main aim of our system is to synthesis voice as similar as possible to the voice provided by the real male speaker. In the front-end, we convert the Chinese character sequences to Pinyin sequences with tone and prosody annotation. In the back-end, the Tacotron 2 model is adapted for predicting spectrogram features. Then, the predicted spectrograms are used to generate 16-bit speech waveforms by Griffin-lim algorithm.
BibTeX
@inproceedings{zhang2019tju,
  author = {Ju Zhang and Shaotong Guo and Cheng Gong and Shuaiting Chen and Yuguang Wang and Longbiao Wang and Wei Zou and Xiangang Li},
  title = {{The TJU-Didi-Huiyan system for Blizzard Challenge 2019}},
  booktitle = {Blizzard Challenge},
  year = {2019},
  url = {https://openreview.net/forum?id=tKsjV11ZDb},
}
Figure