The I2R-NWPU-NUS Text-to-Speech System for Blizzard Challenge 2018

Jinba Xiao, Shan Yang, Mingyang Zhang, Berrak Sisman, Dongyan Huang, Lei Xie, Minghui Dong, Haizhou Li

TL;DR

The I2R-NWPU-NUS team presents their text-to-speech (TTS) system for the Blizzard Challenge 2018, moving from unit selection to a statistical parametric approach using Bidirectional Long Short-Term Memory (BLSTM) recurrent neural networks. This system models acoustic parameters and phone durations, demonstrating improved synthesized speech quality compared to a DNN baseline.

What Problem Does This Paper Address?

Creating natural-sounding synthetic speech from text is a long-standing challenge in speech technology. Traditional methods often struggle with prosody (intonation, rhythm, stress) and can sound robotic. The Blizzard Challenge aims to push the boundaries of TTS by requiring systems to synthesize speech from a single speaker's audiobook data, with a focus on naturalness and intelligibility. This paper addresses the challenge of generating high-quality, natural-sounding speech by exploring advanced deep learning techniques for statistical parametric TTS.

Methods

The core of the I2R-NWPU-NUS system is a statistical parametric approach leveraging deep neural networks. Specifically, they employ Bidirectional Long Short-Term Memory (BLSTM) recurrent neural networks (RNNs) for both modeling acoustic parameters and phone durations.

  • Acoustic Modeling: BLSTM-RNNs are used to predict frame-level acoustic features, including mel-generalized cepstral coefficients (MGC), band aperiodicity (BAP), fundamental frequency (LF0), and voicing/unvoicing (V/UV) flags. Dynamic features (∆ and ∆²) are also incorporated to ensure smooth parameter trajectories.
  • Duration Modeling: A separate BLSTM-RNN models phone durations, predicting them across 5 states per phoneme for finer control.
  • Feature Extraction: The Festival frontend toolkit is used for linguistic feature extraction, and the HTS toolkit for state-level alignment. Acoustic features are extracted using the WORLD vocoder.
  • Parameter Generation: The Maximum Likelihood Parameter Generation (MLPG) algorithm is used to generate smooth acoustic parameter trajectories from the predicted features.
  • Synthesis: The WORLD vocoder is employed to convert the generated acoustic parameters into the final speech waveform.

The paper also briefly discusses experiments with Mixture Density Networks (MDNs) and end-to-end TTS models (Tacotron), noting that the BLSTM approach yielded better results in their experiments for this challenge, particularly in avoiding over-smoothed trajectories and maintaining better energy. They also investigated a reference encoder module to explicitly model prosody, which was integrated with the BLSTM architecture.

Datasets

  • Training Data: The system was trained on approximately 6.5 hours of speech data from professionally produced children's audiobooks, provided by the Blizzard Challenge organizers. This data included cleaned-up text, sentence-level alignments, segmented waveforms, and extracted linguistic/acoustic features.
  • Evaluation Data: Synthesized speech was evaluated on news items, new audiobooks, and semantically unpredictable sentences (SUS), some of which were from previous years.

Key Insights and Ideas

  • BLSTM as a Robust Baseline: The paper highlights BLSTM-RNNs as a powerful and stable choice for statistical parametric TTS, outperforming simpler DNN baselines and even showing promise compared to more complex end-to-end models in their specific experimental setup.
  • Importance of Dynamic Features: Incorporating dynamic features (∆ and ∆²) alongside static acoustic features is crucial for generating smooth and natural-sounding speech, preventing abrupt changes in the synthesized output.
  • Prosody Modeling is Key: The exploration of a reference encoder module underscores the importance of capturing and transferring prosodic information (intonation, rhythm) for more expressive and human-like speech.
  • Trade-offs in End-to-End Models: While end-to-end models like Tacotron offer potential for simplification, the paper points out their sensitivity to training data and potential difficulties in learning accurate alignments, especially when input and output lengths differ significantly.

Key Results

The paper reports that their BLSTM-based statistical parametric TTS system showed improvement compared with the results of the DNN baseline system in listening tests. While specific quantitative metrics for their system's performance are presented in figures (e.g., MOS scores for overall impression, naturalness, and similarity), the text focuses on the comparative improvement. Figure 4 shows their system (labeled "G") achieving competitive MOS scores for overall impression of audiobook paragraphs. Figure 5 shows similar strong performance in naturalness MOS. Figure 6 indicates good similarity scores. Figure 7 shows their Word Error Rate (WER) in the SUS testing, which decreased compared to their previous year's system.

Limitations and Future Work

  • WaveNet Mismatch: The authors encountered issues when attempting to use WaveNet as a vocoder, experiencing significant noise and degraded quality due to a mismatch between ground truth and predicted features. They ultimately reverted to the WORLD vocoder. This suggests that careful integration and training are needed for advanced vocoders.
  • End-to-End Model Sensitivity: The paper notes that end-to-end systems like Tacotron are highly sensitive to the training dataset and can struggle with learning good alignments, especially with complex prosodic variations.
  • Future Work: The authors mention that advanced TTS with WaveNet will be their focus in the near future, implying they aim to overcome the integration challenges. They also suggest that a more robust attention mechanism is required for end-to-end systems to handle the diverse prosodic characteristics present in audiobook data.

Summary updated

Abstract

This paper presents I2R-NWPU-NUS team’s text-to-speech system to Blizzard Challenge 2018. Instead of using unit selection based concatenative speech sysnthesis previous years. we adopt the general deep neural network (DNN) statistical parametric method to synthesize the speech. The frame level acoustic parameters and phone duration are modelled using bidirectional long short-term memory (BLSTM) recurrent neural networks (RNNs). For duration model, 5 states of phone duration are used to predict the duration of each phoneme. Finally, the predicted acoustic parameters (MGC, LF0, V/UV) are taken as inputs to WORLD vocoder to generate the synthetic speech. The listening tests show improvement compared with the results of DNN baseline system.
BibTeX
@inproceedings{xiao2018i,
  author = {Jinba Xiao and Shan Yang and Mingyang Zhang and Berrak Sisman and Dongyan Huang and Lei Xie and Minghui Dong and Haizhou Li},
  title = {{The I2R-NWPU-NUS Text-to-Speech System for Blizzard Challenge 2018}},
  booktitle = {Blizzard Challenge},
  year = {2018},
  doi = {10.21437/blizzard.2018-4},
}

Loading related content...

Figure