TL;DR
This paper introduces the use of deep, bi-directional Recurrent Neural Networks (RNNs), specifically Long Short-Term Memory (LSTM) networks, for predicting prosody in text-to-speech (TTS) systems. By leveraging RNNs' ability to capture long-term dependencies in sequential data, they achieve improved objective metrics and perceptual quality compared to state-of-the-art Deep Neural Networks (DNNs).
What Problem Does This Paper Address?
Generating natural-sounding speech in text-to-speech (TTS) systems is a complex challenge. While current systems can reproduce basic speech sounds, achieving natural prosody—the rhythm, stress, and intonation of speech—remains difficult. Existing models, like DNNs, often rely on fixed-size context windows, which struggle to capture the intricate interplay of short-term and long-term contextual factors that influence how we speak. This limitation can lead to prosody that sounds discontinuous or lacks the expressiveness found in human speech. The paper aims to address this by modeling prosody prediction as a sequence-to-sequence problem, where the system needs to understand dependencies across longer stretches of text to predict prosodic features accurately.
Methods
The authors propose using deep, bi-directional Recurrent Neural Networks (RNNs) for prosody prediction. Unlike traditional DNNs that process input features independently, RNNs maintain an internal state that is updated over time, allowing them to learn from sequential data and capture temporal dependencies.
Key methodological points include:
- Bi-directional RNNs: These networks process the input sequence in both forward and backward directions, allowing them to consider both past and future context when making a prediction at any given time step.
- Long Short-Term Memory (LSTM): To overcome the challenges of training deep recurrent networks (like vanishing gradients), the paper employs LSTM cells. LSTMs are specifically designed with "gates" (input, forget, and output) and a "Constant Error Carousel" (CEC) to effectively learn and retain information over long sequences. The paper uses an extended LSTM cell with "peephole connections" for enhanced performance.
- Deep Architecture: The RNNs are structured in multiple layers, both temporally (deep in time) and across layers (deep in layers), enabling them to learn increasingly complex representations of the input.
- Weighted Regression: Prosody prediction is framed as a weighted regression problem. The model predicts seven target statistics (mean and variance for F0, A-logF0, AA-logF0, and mean for log duration) required by a standard HMM parametric synthesizer. A weighting scheme is used to focus learning on relevant streams and observations, particularly for voiceless sounds or silences where certain prosodic features might not be directly applicable.
- Baseline DNN: A 3-layer DNN with sigmoid hidden units and a linear output layer, trained using a two-pass approach (unsupervised pre-training followed by supervised fine-tuning), serves as the baseline for comparison.
Datasets
The paper does not mention any newly introduced benchmarks. The evaluation was performed on two distinct corpora of North-American English speech from female speakers:
- Voice F1: Approximately 7 hours of speech recorded in a business-news style.
- Voice F2: Approximately 10 hours of speech from various genres and domains.
Each corpus was divided into training (80%), development (10%), and test (10%) sets. The speech was phonetically aligned, and F0 contours were extracted at a 5ms frame rate.
Key Insights and Ideas
- Long-term dependencies are crucial for natural prosody: The paper strongly suggests that the limitations of fixed-context models (like DNNs) in capturing long-term prosodic patterns are a significant bottleneck for naturalness.
- RNNs excel at sequential modeling for prosody: The inherent ability of RNNs to model temporal dependencies makes them a natural fit for prosody prediction, where context evolves over time.
- LSTMs are essential for training deep RNNs: The success of deep RNNs hinges on architectures like LSTMs that can effectively manage information flow and prevent gradient issues over long sequences.
- Improved variance prediction matters: A key finding is that RNNs not only match DNNs in fitting the mean prosodic contours but also significantly improve the prediction of prosodic variance. This is highlighted as potentially having perceptual implications for expressiveness.
- Perceptual gains validate objective improvements: The listening tests confirm that the objective improvements translate into a better perceived prosody, bridging a significant portion of the gap between the baseline DNN and "perfect" natural prosody.
Key Results
- Objective Metrics:
- F0 Mean-Square Error (MSE): RNNs achieved a relative reduction of approximately 6% for F1 and 6.23% for F2 compared to DNNs.
- F0 Variance (NVAR): RNNs showed a relative increase of about 14% for F1 and 18.08% for F2 in predicted variance compared to DNNs, indicating a better capture of prosodic range.
- Cross-Correlation (XCORR): RNNs consistently outperformed DNNs in XCORR for most prosodic streams.
- Perceptual Evaluation (Mean Opinion Scores - MOS):
- For Voice F1, the RNN achieved a MOS of 3.56, compared to 3.49 for the DNN baseline, closing 35% of the gap to transplanted natural prosody (3.69).
- For Voice F2, the RNN achieved a MOS of 3.84, compared to 3.76 for the DNN baseline, closing 47% of the gap to transplanted natural prosody (3.93).
Limitations and Future Work
- Limitations:
- The paper acknowledges that even with RNNs, there's still a gap between the predicted prosody and perfectly natural prosody (as evidenced by the MOS scores).
- The authors note that the DNN model provides a strong baseline, and the question of how much improvement can be gained solely by improving prosody prediction is explored.
- Future Work:
- The paper implicitly suggests that further exploration of deep recurrent architectures and their application to other aspects of speech synthesis (like spectral envelope prediction) could be beneficial.
- The authors' focus on improving prosody prediction suggests that future work might involve integrating these advanced prosody models into full TTS systems to assess their impact on overall speech quality.
Summary updated