TL;DR
This paper presents the IIM-USTC speech synthesis system for the Blizzard Challenge 2019, focusing on improving HMM-based unit selection with novel LSTM-RNN and DNN-based acoustic modeling techniques. While achieving similar performance to their previous year's system, they highlight the continued advantage of unit selection for speaker similarity, even as parametric systems show gains in naturalness.
What Problem Does This Paper Address?
The paper tackles the challenge of building high-quality, natural-sounding speech synthesis systems. Specifically, it focuses on the Blizzard Challenge task of creating a system using an 8-hour Chinese male talkshow audio corpus. Traditional unit selection systems, while good at preserving speaker identity, can struggle with naturalness due to limitations in modeling complex context dependencies and long-term prosody. Statistical parametric speech synthesis (SPSS) systems offer more flexibility but are often limited by vocoder quality. The authors aim to bridge this gap by enhancing HMM-based unit selection with advanced neural network techniques to improve acoustic modeling and context representation.
Methods
The IIM-USTC system builds upon a conventional HMM-based unit selection framework. Key improvements include:
- LSTM-RNN based Acoustic Model: To capture long-term dependencies in sequential data, an LSTM-RNN model is employed. The hidden layer of this model is used to extract "context embedding features," which are then incorporated into the unit selection criterion. This addresses the limitation of traditional HMMs in modeling complex context dependencies.
- DNN based Unit Embedding Model (Unit2Vec): A DNN, termed Unit2Vec, is introduced to learn fixed-length vector representations for each phone unit. These "unit embeddings" capture the acoustic characteristics of candidate units, providing a more robust feature for unit selection. The model is trained by minimizing the mean square error between predicted and natural acoustic features.
- Enhanced Unit Selection Criterion: The synthesis phase incorporates both frame-level acoustic features and the newly derived context embedding features and unit embeddings. The selection criterion is a weighted combination of costs derived from HMM-based models, KLD divergence of context embeddings, and Euclidean distance of phone-level unit embeddings. Dynamic programming is used to find the optimal unit sequence.
Datasets
- Training and Evaluation Data: An 8-hour Chinese male talkshow audio corpus was used for training and evaluation, as specified by the Blizzard Challenge 2019.
- Benchmark Systems: The evaluation included 1 benchmark system (Merlin), the USTC-iflytek system, and the authors' system (labeled 'Z'), alongside natural speech.
Key Insights and Ideas
- Neural Networks for Context: The paper strongly advocates for using neural networks (LSTM-RNN and DNNs) to model complex context dependencies, which are crucial for improving the naturalness and expressiveness of unit selection systems.
- Unit Embeddings as Acoustic Signatures: The Unit2Vec model demonstrates the effectiveness of learning fixed-length vector representations for phonetic units, providing a compact and informative signature for unit selection.
- Unit Selection Still Competitive for Similarity: Despite advancements in parametric systems, the results suggest that unit selection methods still hold an advantage in preserving speaker similarity, a critical aspect for many speech synthesis applications.
- Parametric Systems Gaining on Naturalness: The paper notes a significant finding: a parametric speech synthesis system (System M) outperformed the best unit selection system in naturalness, indicating a rapid advancement in SPSS.
Key Results
- Naturalness: The IIM-USTC system ranked fourth in naturalness tests, achieving a Mean Opinion Score (MOS) of 4.0. However, a parametric system (M) was significantly better.
- Similarity: The system achieved the equal highest speaker similarity among submitted systems, ranking second overall in the similarity test. The difference in similarity compared to the best system (M) was not significant.
- Intelligibility: The system ranked 10th in intelligibility tests, with a Pinyin error rate (PER) of 12.4%. The authors acknowledge that unit selection systems are not superior in intelligibility, especially for talkshow-like corpora.
- Overall Performance: The authors conclude that their system performed at the same level as their previous year's submission, but other systems showed better performance.
Limitations and Future Work
- Intelligibility: The paper explicitly states that the unit selection system is not superior in intelligibility, particularly for the talkshow audio corpus. This suggests that future work might explore methods to improve phonetic accuracy.
- Performance Gap: While the system performed comparably to their previous year's work, it was outperformed by other systems in the challenge, indicating room for further optimization.
- Parametric System Advancement: The success of parametric systems in naturalness suggests that future research might focus on hybrid approaches or further advancements in SPSS to leverage their strengths while mitigating weaknesses.
- Computational Complexity: Although not a primary focus, the use of LSTM-RNN and DNNs implies significant computational resources are needed for training. The paper mentions using a recurrent layer with LSTMP architecture to reduce computational cost.
Summary updated