TL;DR
This paper introduces a novel "Speech BERT" model to extract prosody embeddings from speech segments, which are then used to enhance the prosody of synthesized speech in neural Text-to-Speech (TTS) systems. By leveraging large-scale pre-training, this approach significantly improves the naturalness and expressiveness of generated speech, outperforming existing methods in both objective and subjective evaluations.
What Problem Does This Paper Address?
Neural Text-to-Speech (TTS) systems have made remarkable progress, particularly with the advent of end-to-end models like Transformer TTS. However, a persistent challenge lies in accurately capturing and reproducing the nuances of human speech prosody – the rhythm, stress, and intonation that convey meaning and emotion. While current TTS models can map text to acoustic features, they often struggle with high-level expression and rich prosodic variability, leading to synthesized speech that can sound monotonous or unnatural. Existing methods for controlling prosody, such as style tokens or prosody transfer, often rely on limited data or struggle with out-of-domain scenarios. This paper aims to bridge this gap by using a pre-trained model to learn robust prosodic representations from extensive speech data, thereby improving the prosody of synthesized speech.
Methods
The core innovation is the development of a Speech BERT model specifically designed for TTS. This model is inspired by the success of BERT in Natural Language Processing (NLP) and Automatic Speech Recognition (ASR).
Here's a breakdown of the key techniques:
- Speech BERT Architecture: The Speech BERT model is built upon the Transformer TTS framework but with a crucial addition: a speech encoder block. This encoder is designed to learn prosodic attributes from speech segments.
- Masked Segment Reconstruction: Similar to BERT's masked language modeling, the Speech BERT model is trained by masking segments of the input mel-spectrogram sequence and then reconstructing them. The model is conditioned on the text sequence and the surrounding speech context.
- Prosody Extraction: The Speech BERT encoder is trained to reconstruct the content of the masked speech segment based on the text. Crucially, it's hypothesized that the encoder also learns other acoustic attributes, including prosody, from the context. The embeddings generated by this encoder are thus treated as prosody representations.
- Dynamic Embedding Integration: The extracted Speech BERT prosody embeddings are integrated dynamically into the Transformer TTS decoder. Specifically, the embedding from the previous speech segment is used to predict the current segment. This dynamic approach allows for fine-grained, segment-level prosody modeling.
- Mask Template Generation: To train the Speech BERT model, a mask template is generated using successive Dynamic Time Warping (DTW) on averaged phone segments from a diverse set of speech recordings. This ensures the template is in the acoustic space, represents time-varying distributions, and is generalizable to pronunciation variations.
Datasets
The paper utilizes several datasets for training and evaluation:
- Speech BERT Training:
- LJSpeech
- VCTK
- LibriTTS (both "other" and "clean" subsets)
- Note: For Speech BERT training, only recordings shorter than 10 seconds were used due to GPU memory constraints.
- Transformer TTS Training:
- Internal single-speaker data (19.08 hours)
- Public LJ speaker data (LJSpeech)
- Internal multi-speaker data (30 speakers, 247.45 hours)
- Evaluation:
- The same datasets used for Transformer TTS training were used for evaluation.
- Out-of-domain evaluation was performed using news domain sentences from CNN.
Key Insights and Ideas
- Pre-training for Prosody: The paper demonstrates that pre-training a BERT-like model on a large speech corpus is highly effective for learning robust prosody representations, which can then be transferred to improve TTS. This is a significant departure from traditional TTS methods that learn prosody from scratch on limited in-domain data.
- Segment-Level Prosody is Key: By extracting embeddings from syllable-length segments and using them dynamically, the model captures prosodic variations at a finer granularity than utterance-level methods. This "multi-scale variability modeling" is crucial for naturalness.
- Out-of-Domain Generalization: The pre-trained Speech BERT embeddings provide auxiliary information that helps the TTS model generalize better to unseen sentences or styles, significantly improving performance in out-of-domain scenarios.
- Complementary to Utterance-Level Prosody: The segment-level prosody captured by Speech BERT complements existing utterance-level prosody modeling, leading to a more comprehensive and natural output.
Key Results
- Objective Evaluation: Objective distortions (mean square error) for F0, energy, and duration between synthesized and recorded speech were reduced when using the Speech BERT embedding, particularly on the training set.
- Subjective Evaluation (MOS): For single-speaker TTS, the MOS score of the synthesized speech with Speech BERT embedding was closer to the ground truth recording (4.46 vs. 4.42 for w/o).
- Subjective Evaluation (Preference Tests):
- Across single-speaker, multi-speaker, and the LJ speaker datasets, the proposed method with Speech BERT embedding was consistently preferred over the baseline Transformer TTS.
- In out-of-domain tests (news domain), the preference for the Speech BERT enhanced TTS was even more pronounced, highlighting its generalization capabilities.
Limitations and Future Work
- Limitations:
- The paper notes that for Speech BERT training, only recordings shorter than 10 seconds were used due to GPU memory limitations. This might limit the full potential of the pre-training if longer, more context-rich utterances were available.
- While the model improves prosody, the paper doesn't explicitly detail how it handles very subtle emotional nuances or complex prosodic styles beyond general expressiveness.
- Future Work:
- The authors suggest that the Speech BERT embedding can be further explored for other TTS-related tasks, such as style transfer or speaker adaptation.
- Investigating the use of longer speech segments or full utterances for Speech BERT pre-training could be a promising direction.
- Exploring different masking strategies or pre-training objectives for the Speech BERT model might yield further improvements.
Summary updated