TL;DR
This paper introduces a novel approach to speech prosody control by combining weighted neural network ensembles with feature relevance determination. This method significantly improves the accuracy of predicting phone duration and fundamental frequency (Fo) compared to existing state-of-the-art models, leading to more natural-sounding synthetic speech.
What Problem Does This Paper Address?
Generating natural-sounding synthetic speech requires accurately predicting prosodic features like pitch (Fo), duration, and intensity. Current text-to-speech (TTS) systems often struggle with this due to limitations in statistical models, such as low prediction accuracy, poor generalization to unseen linguistic contexts, and sensitivity to errors in the training data. Specifically, models like CART, MARS, and even standard Artificial Neural Networks (ANNs) have drawbacks when dealing with limited prosody corpora. This paper aims to overcome these limitations by developing a more robust and accurate statistical model for prosody control.
Methods
The core of this work lies in two key techniques:
- Weighted Neural Network Ensembles: Instead of relying on a single model, the authors combine multiple independently trained neural networks (base learners). The final prediction is a weighted average of the individual predictions, where each network's contribution is determined by its performance on the training data. This ensemble approach enhances accuracy, generalization, and robustness against outliers.
- Feature Relevance Determination: To combat the "curse of dimensionality" and improve model interpretability, especially with limited data, the paper employs a method to identify and remove less relevant input features. This is achieved by extending the "Optimal Brain Surgeon" (OBS) algorithm, specifically the "Unit-OBS" variant, which prunes entire input units based on their contribution to the network's performance. This allows for more efficient training and better generalization.
For the base learners, feed-forward ANNs were used for segment duration prediction, and recurrent neural networks (RNNs) were used for Fo contour prediction. The ensemble aggregation method chosen was Weighted Bagging (W-Bagging), which uses cross-validation to create training subsets.
Datasets
The models were evaluated on three prosody corpora:
- German Male: Approximately 30 minutes of speech from a professional male speaker, with a test set of 55 sentences.
- German Female: Approximately 30 minutes of speech from a professional female speaker, with a test set of 44 sentences.
- French Female: Approximately 30 minutes of speech from a professional female speaker, with a test set of 48 sentences.
The German male corpus used manually segmented and labeled sentences for both training and testing. For the German and French female corpora, training sentences were automatically segmented, while test sentences were manually segmented and labeled.
Key Insights and Ideas
- Ensembles are Powerful: Combining multiple ANNs significantly boosts prediction accuracy and robustness, outperforming single ANN models.
- Feature Selection is Crucial: For limited datasets and complex prosody modeling, identifying and removing irrelevant features is not just beneficial but essential for good performance and interpretability.
- Weighted Ensembles Outperform Simple Averaging: Using a weighted average of base learner predictions, where weights reflect individual performance, is more effective than a simple arithmetic mean.
- Exponential Weighting is Robust: The exponential weighting function for ensemble members proved more robust than a potential weighting function, especially when dealing with potential overfitting.
- Ensembles Complement Feature Pruning: The combination of ensemble learning and feature relevance determination is particularly effective when dealing with small training datasets, a common challenge in prosody modeling.
Key Results
The weighted neural network ensemble models demonstrated significant improvements over existing methods:
- Duration Prediction: Achieved a 12% improvement compared to the best duration model (CART-based) and 14.6% to 16.1% better than ANN models. The improvement over MARS models was around 11.5%.
- Fo Prediction: Showed a 24% improvement compared to the best Fo model (RNN-based) and performed about 9% better than RNN models with optimized input factors.
- Perceptual Evaluation: A listening experiment indicated that approximately 87.5% of synthetic German prosody contours and 92.5% of synthetic French prosody contours were indistinguishable from natural prosody.
Limitations and Future Work
- Limited Corpus Size: While the proposed method performs well with limited data, the authors acknowledge that prosody corpora are generally small, which is a primary motivation for their approach. Further improvements might still benefit from larger, more diverse datasets.
- Computational Cost: Ensemble methods, especially those involving extensive feature selection and multiple base learners, can be computationally intensive. The paper mentions W-Bagging was chosen over W-SECA due to lower computational cost, but optimization is still a consideration.
- Future Work: The paper implicitly suggests that further exploration of different ensemble aggregation techniques and more sophisticated feature relevance determination methods could yield even better results. The success of this approach also opens doors for applying similar techniques to other aspects of speech synthesis.
Summary updated