TL;DR
This paper introduces an advanced unsupervised method (A-UJPLM) to jointly model Mandarin speech's pitch, duration, and energy, leading to more interpretable prosodic models. This improved modeling is then successfully applied to generate more natural-sounding prosody for Text-to-Speech (TTS) systems.
What Problem Does This Paper Address?
Creating natural-sounding synthesized speech, particularly in Mandarin, is a significant challenge for Text-to-Speech (TTS) systems. A crucial aspect of naturalness is prosody – the rhythm, stress, and intonation of speech. Traditional prosody generation methods often fall into two categories: direct modeling, which is hard to interpret and modify, or hierarchical approaches, which can suffer from subjective and time-consuming manual labeling of prosodic features. This paper aims to bridge this gap by developing an unsupervised method that can automatically learn and label prosodic information, making it more interpretable and applicable to TTS.
Methods
The core of this work is the Advanced Unsupervised Joint Prosody Labeling and Modeling (A-UJPLM) method. Building upon a previous Unsupervised Joint Prosody Labeling and Modeling (UJPLM) approach, A-UJPLM extends the modeling to include not only syllable pitch contour but also syllable duration and energy level.
Key aspects of the A-UJPLM method include:
- Joint Modeling: Simultaneously learning models for pitch, duration, and energy. This is a significant advancement over previous methods that might focus on individual features.
- Prosodic Hierarchy and Tags: The model operates within a defined prosodic hierarchy (syllable, prosodic word, prosodic phrase, breath/prosodic phrase group) and uses prosodic tags to represent these structures. New "prosodic state" tags for duration and energy are introduced to capture high-level prosodic influences.
- Additive Affecting Patterns (APs): The model decomposes prosodic variations into additive components, including the effects of lexical tones, prosodic states, and coarticulation. This decomposition aids in interpretability.
- Unsupervised Learning: The method learns these models and labels from unlabeled speech data, eliminating the need for manual annotation.
- Break Prediction and Prosodic Feature Prediction: For TTS application, the process is divided into two steps:
- Break Prediction: Predicting the sequence of prosodic breaks (e.g., boundaries between words, phrases) using a break-syntax model.
- Prosodic Feature Prediction: Generating syllable pitch contour, duration, and energy level using the learned prosodic models and the predicted break sequence.
Datasets
- Sinica Treebank corpus: Used for both training and evaluation of the A-UJPLM method. This corpus is noted as being read Mandarin speech.
- The paper mentions an unlabeled read Mandarin speech database containing 425 utterances with 56,237 syllables, divided into training (379 utterances) and testing (46 utterances) sets.
Key Insights and Ideas
- Interpretability through Unsupervised Learning: The A-UJPLM method provides linguistically meaningful and interpretable prosodic models without manual labeling, which is a significant advantage over traditional methods.
- Decomposition of Prosodic Variation: The additive affecting patterns (APs) allow for a clear understanding of how different factors (tones, phrase structure, coarticulation) contribute to the overall prosody.
- Importance of Break Prediction: The study highlights that accurate prediction of prosodic breaks is crucial for generating natural-sounding prosody. Errors in break prediction directly lead to significant errors in prosodic feature generation.
- Synergistic Modeling: Jointly modeling pitch, duration, and energy leads to better overall prosodic representation and generation compared to modeling features independently.
Key Results
- Meaningful Prosodic Tags: Experimental results showed that the automatically labeled prosodic tags were linguistically meaningful.
- Improved Prosody Generation: The A-UJPLM-based TTS system demonstrated good performance, with most predicted prosodic features closely matching their original counterparts.
- Break Prediction Accuracy: The break prediction model achieved reasonable accuracy, with high rates for major breaks (B1, B4) and moderate rates for others. However, minor breaks (B2-1, B2-2, B2-3) were more challenging to predict accurately.
- Prosodic Feature Prediction Accuracy: Total Residual Errors (TREs) for pitch, duration, and energy prediction were reported. For example, using predicted breaks, TREs were 42.7% for pitch, 46.2% for duration, and 36.0% for energy. Using correct break labels significantly improved these results, reducing TREs to 39.1%, 41.7%, and 33.3% respectively.
Limitations and Future Work
- Minor Break Prediction: The paper acknowledges that predicting minor breaks (B2-1, B2-2, B2-3) remains a challenge and suggests that incorporating more sophisticated linguistic features, such as syntactic phrase or tree information, could improve their prediction.
- Break Prediction Errors Impact: The study clearly demonstrates that errors in break prediction significantly degrade the quality of prosodic feature generation. Future work should focus on improving break prediction accuracy.
- Further Investigation: The authors suggest that further investigation into break prediction is essential for improving prosodic feature generation.
Summary updated