SJTU Entry in Blizzard Challenge 2019

Bo Chen, Kuan Chen, Zhijun Liu, Zhihang Xu, Songze Wu, Chenpeng Du, Muyang Li, Sijun Li, Kai YuBlizzard Challenge 2019 · OpenReview

TL;DR

This paper details the SJTU team's entry for the Blizzard Challenge 2019, focusing on a Tacotron-WaveNet architecture adapted for a noisy, mixed-language Chinese-English speech corpus. Despite challenges with vocoder quality on corrupted data, the team explored various techniques for data preprocessing, prosody modeling, and bandwidth extension to improve synthetic speech.

What Problem Does This Paper Address?

The Blizzard Challenge 2019 presented a unique hurdle: a relatively small (8-hour) corpus of Chinese-English talk-show audio that was heavily corrupted and noisy, likely recorded with mobile phones in uncontrolled environments. This "found speech" data is far from the clean, studio-quality recordings typically used to train state-of-the-art text-to-speech (TTS) systems like Tacotron and WaveNet. The core problem is adapting these powerful models to perform well on such degraded and limited data, specifically addressing issues like noise, prosody (intonation, rhythm), and the mixed-language nature of the corpus.

Methods

The SJTU team's approach was a four-part system:

  • Data Pre-processing: This was crucial given the noisy input. They employed Voice Activity Detection (VAD) to segment audio, followed by a multi-speaker Hidden Markov Model (HMM) based Automatic Speech Recognition (ASR) system to transcribe segments. These transcriptions were then refined against full sentence transcriptions. For mixed-language input, text was converted into phoneme sequences, handling Chinese syllables and English phonemes separately.
  • Spectrogram Model (Tacotron-based): The core of their acoustic modeling used a Tacotron-like architecture. To mitigate error accumulation during inference, they modified the standard Tacotron by abandoning direct mel-spectrogram decoding. Instead, Tacotron's attention mechanism was used to generate frame-level embeddings, which were then fed into stacked bidirectional LSTMs to predict the mel-spectrogram. This approach aimed to leverage the strengths of statistical parametric speech systems (SPSS) for better prosody. They also explored multi-speaker training with speaker and environment embeddings to handle variations in the corpus.
  • WaveNet Vocoder: A 10-bit autoregressive WaveNet was used for waveform generation. To adapt it to the specific LZY speaker and corpus, they employed a multi-speaker WaveNet initialization followed by speaker-specific adaptation, including learning a dedicated speaker embedding.
  • Speech Bandwidth Extension (BWE): Recognizing that the target audio had more noise in high frequencies, their acoustic model targeted a 16kHz sample rate. A separate waveform modeling neural network (based on Hierarchical Recurrent Neural Networks - HRNN) was then used to upsample the 16kHz synthesized speech to 32kHz, aiming to restore some of the lost high-frequency detail. This BWE model was trained on a large, clean, multi-speaker dataset.

Datasets

  • Training Data:
    • LZY Corpus: 8 hours of mixed Mandarin and English talk-show audio from a single male speaker. This data was heavily corrupted and noisy.
  • Auxiliary Clean Corpus: Used for multi-speaker Tacotron training to improve prosody modeling, particularly for tones.
  • Multi-speaker BWE Dataset: Approximately 2000 hours of 32kHz audiobook speech from 25 speakers (mostly male, Chinese) used to train the bandwidth extension model.
  • Evaluation Data: Not explicitly detailed, but the systems were evaluated on synthetic speech quality (naturalness, similarity, phone error rate) using provided test scripts.

Key Insights and Ideas

  • Tacotron as an Embedding Model: Modifying Tacotron to act primarily as an embedding generator, feeding into a separate LSTM-based post-network for mel-spectrogram prediction, proved more robust for noisy data than standard Tacotron.
  • Environment Embeddings for Noise Control: Using environment embeddings in the Tacotron model helped stabilize synthesis and control noise conditions, preventing drift over longer utterances.
  • Bandwidth Extension for Corrupted High Frequencies: Applying BWE to upsample synthesized speech from 16kHz to 32kHz was found to have a positive effect on speech quality, particularly for the corrupted high-frequency bands.
  • Vocoder Challenges with Noisy Spectrograms: The biggest takeaway was the difficulty in training a high-quality WaveNet vocoder when conditioned on mel-spectrograms predicted from a noisy, corrupted corpus. This significantly impacted the overall system performance.
  • Prosody vs. Naturalness Trade-off: The team noted that while their methods aimed to improve prosody, the ultimate evaluation metric was subjective naturalness, and the vocoder's noise level played a significant role.

Key Results

  • The team's entry achieved a score that had a "gap" to the best systems in the challenge.
  • The primary reason cited for this gap was the poor performance of the WaveNet vocoder on the predicted mel-spectrograms, resulting in more noise than Griffin-Lim vocoded speech.
  • The multi-speaker Tacotron with environment embeddings successfully improved tone modeling.
  • Bandwidth extension was observed to have a positive effect on synthetic speech quality.
  • Experiments with GAN-based post-filters for spectrogram enhancement did not yield improvements after vocoding.

Limitations and Future Work

  • WaveNet Vocoder Quality: The most significant limitation was the inability to train a sufficiently good WaveNet vocoder on the predicted mel-spectrograms from the corrupted corpus. This meant that improvements in other parts of the system (like prosody modeling) did not translate into significant overall gains after vocoding.
  • Data Scarcity: The limited 8-hour corpus was a major constraint, necessitating the use of auxiliary data and multi-speaker models.
  • Subjective Evaluation of Vocoders: The performance analysis highlighted a preference split between younger and older listeners for different vocoder outputs (WaveNet vs. a mixed vocoder), suggesting that different age groups prioritize different aspects of speech quality.
  • Future Work: The paper implicitly suggests that improving the robustness of the vocoder to noisy or mismatched spectrograms is a critical area for future research. Further investigation into more effective ways to model prosody and handle corrupted input data would also be beneficial.

Summary updated

Abstract

This paper presents the techniques that were used in sjtu-tts entry in Blizzard Challenge 2019. The main architecture is Tacotron with WaveNet vocoder. The corpus in BC2019 is 8 hours audios from a Chinese male speaker with mixed Mandarin and English speech. The audios and transcriptions are found on the Internet with heavily corruption and noise. To deal with the corpus, our system is divided into 4 parts, data preprocessing, spectrogram model, WaveNet vocoder and speech bandwidth extension. The WaveNet vocoder is more relative to the speech quality and the spectrogram model is more relative to the prosody(pitch and duration). We didn’t successfully train a good WaveNet vocoder for the predicted mel-spectrogram. Thus, some useful techniques in other parts have no significant improvement after WaveNet vocoding. These attempts which were not included in the final submission are also analyzed.
BibTeX
@inproceedings{chen2019sjtu,
  author = {Bo Chen and Kuan Chen and Zhijun Liu and Zhihang Xu and Songze Wu and Chenpeng Du and Muyang Li and Sijun Li and Kai Yu},
  title = {{SJTU Entry in Blizzard Challenge 2019}},
  booktitle = {Blizzard Challenge},
  year = {2019},
  url = {https://openreview.net/forum?id=bZ062tyRGY},
}
Figure