The NLPR Speech Synthesis entry for Blizzard Challenge 2017

Jianhua Tao, Ruibo Fu, Yibin Zheng, Zhengqi Wen, Ya Li, Biu LiuBlizzard Challenge 2017 · OpenReview

TL;DR

This paper presents the NLPR team's entry for the Blizzard Challenge 2017, focusing on a BiLSTM-guided unit selection speech synthesis system trained on children's audiobooks. They introduce novel improvements in unit selection strategies, particularly in how acoustic and contextual costs are calculated, to better handle expressive speech and improve naturalness.

What Problem Does This Paper Address?

The goal of the Blizzard Challenge is to build high-quality, natural-sounding speech synthesis systems. This paper specifically tackles the challenge of creating a system suitable for reading children's audiobooks, which often involves expressive speech and varied character portrayals. Traditional unit selection systems are known for their naturalness but can struggle with the degradation that occurs during parameter extraction and modeling in statistical parametric speech synthesis (SPSS). The authors aim to bridge this gap by combining the strengths of both approaches, using BiLSTM to guide unit selection for improved expressiveness and naturalness.

Methods

The NLPR system is a hybrid approach, combining BiLSTM-based statistical models with unit selection. The core of their method involves:

  • BiLSTM-guided Unit Selection: Instead of relying solely on traditional HMMs, they employ BiLSTM networks for both acoustic and duration modeling. This allows the system to learn more complex patterns in speech.
  • Data Preprocessing: They address the issue of over-expressive utterances in the training data by using an acoustic feature-based model to identify and remove them. They also implement an "auto-labeling" mechanism to categorize utterances by their degree of expressiveness.
  • Improved Unit Selection Strategies:
    • Contextual Cost Calculation: The concatenation cost calculation is adjusted to consider the utterance's context and combines acoustic and lexical features more effectively than a simple weighted linear combination.
  • Target Cost Computation: Since BiLSTMs don't have the concept of "states" like HMMs, the authors investigate and compare four different approaches for calculating the target cost, aiming to better capture the core articulation of units. These include Kullback-Leibler Divergence (KLD), Maximum Likelihood Criterion (LL), and a relative position-based Euclidean Distance (ED).
  • Pre-selection: To speed up the computationally intensive unit selection process, they use a contextual information difference (CID) based pre-selection to narrow down the candidate units.

Datasets

The primary dataset used is the British English Speech Corpus for the Blizzard Challenge 2017, consisting of approximately 6.5 hours of speech data from professionally-produced children's audiobooks, recorded by a single female talker. This corpus includes about 4 hours of pilot data from the previous year's challenge. Sentence-level alignment between text and speech was provided by Toshiba's Cambeidge Research Laboratory.

Key Insights and Ideas

  • BiLSTM for Expressive Speech: The use of BiLSTM networks is a key innovation, allowing the system to better model the nuances of expressive speech found in audiobooks, which is a significant challenge for traditional systems.
  • Context-Aware Cost Calculation: Recognizing that not all utterances are the same, the improved cost calculation that considers utterance context and expression degree is crucial for generating more natural and appropriate speech.
  • Addressing Over-Expressiveness: The explicit effort to identify and remove or label over-expressive utterances is a practical step towards creating a more stable and consistent synthesis system.
  • Investigating Target Cost for BiLSTMs: The exploration of different target cost calculation methods highlights the ongoing research into adapting deep learning models for speech synthesis, particularly when they deviate from traditional HMM frameworks.

Key Results

The paper reports on internal evaluations and the official Blizzard Challenge 2017 results.

  • Internal Evaluation: An internal listening test comparing different concatenation cost calculation approaches showed that combining acoustic parameters (f0, energy, lsp, mfcc) with contextual information (system 4) outperformed other combinations.
  • Blizzard Challenge 2017: The system's performance in the official evaluation (naturalness, similarity, and intelligibility) was described as "average level" compared to the top systems. The authors acknowledge that there is still a gap to be closed, particularly in improving the consistency between concatenated units.

Limitations and Future Work

  • Consistency of Concatenation: The authors explicitly state that improving the consistency between concatenated units is a major area for future work, as it was identified as a key factor affecting listener impressions.
  • Text Analysis Accuracy: The text analysis module, relying on the Festival toolkit, was found to be not always accurate, which can negatively impact subsequent stages of the synthesis pipeline. Improving the accuracy of text analysis is highlighted as a necessary step.
  • Gap to Top Systems: While the system achieved an average performance, there's a clear indication that further research and development are needed to reach the state-of-the-art.

Summary updated

Abstract

The paper describes the CASIA speech synthesis system entry for Blizzard Challenge 2017. About 6.5 hours of speech data from professionally-produced children’s audiobooks is adopted as the training data for the construction this year. Our synthesis system is built based on the BiLSTM guided unit selection and waveform concatenation approaches by using the provided corpus. Different from our previous system, some improvements about unit selection strategies were made to adapt to different types of the utterance. In this paper, the definitions of the acoustic and the contextual parameters, strategies of candidate unit selection, the calculation of costs based different contexts will be introduced and discussed. Finally, the results of the listening test will be presented.
BibTeX
@inproceedings{tao2017nlpr,
  author = {Jianhua Tao and Ruibo Fu and Yibin Zheng and Zhengqi Wen and Ya Li and Biu Liu},
  title = {{The NLPR Speech Synthesis entry for Blizzard Challenge 2017}},
  booktitle = {Blizzard Challenge},
  year = {2017},
  url = {https://openreview.net/forum?id=s0b8aLCzkj},
}
Figure