TL;DR
This paper tackles the challenge of automatically evaluating the naturalness of synthetic speech, a crucial but difficult task for TTS system development. They propose a novel hierarchical neural network approach that leverages both stimulus-level (individual utterance quality) and system-level (overall synthesizer performance) predictions, outperforming traditional methods and showing promise for more objective and efficient TTS tuning.
What Problem Does This Paper Address?
Developing and fine-tuning Text-to-Speech (TTS) systems is hampered by the lack of reliable, objective measures for assessing the naturalness of synthesized speech. While metrics like Mel-Cepstral Distance (MCD) and Root Mean Squared Error (RMSE) of fundamental frequency (F0) exist, they often correlate poorly with human perception. This forces researchers to rely on expensive and time-consuming subjective listening tests, where human listeners rate speech quality. This bottleneck makes iterative system improvement difficult. Existing objective measures often struggle with local degradations (e.g., sudden pitch changes) and global patterns (e.g., prosody over a syllable), which significantly impact perceived naturalness.
Methods
The authors propose a hierarchical prediction framework that combines predictions at two levels:
- Stimulus-level prediction: Aims to predict the quality of individual utterances.
- System-level prediction: Aims to predict the overall performance of a TTS synthesizer.
Instead of directly predicting stimulus-level scores, their hierarchical approach predicts the difference between stimulus-level and system-level scores. This leverages the robustness of system-level predictions, which are generally easier to obtain accurately.
They experiment with two main types of neural networks:
- Feed-forward Neural Networks (NNs): Used for both stimulus and system-level prediction, with various configurations of hidden layers.
- Convolutional Neural Networks (CNNs): Specifically designed to capture local and global degradations by processing speech features as sequences. CNNs use convolutional and pooling layers to extract features at different levels of detail, offering a more nuanced representation than traditional frame-wise analysis.
They also compare these neural network approaches against linear regression (LR) baselines, including an "elastic net" regression. A key aspect is the simultaneous optimization of both stimulus and system-level predictors within the hierarchical NN framework.
Datasets
The primary dataset used is the publicly available data from six years of the Blizzard Challenge (BC) [2008-2013]. This dataset is unique because it includes:
- Synthetic speech generated by multiple TTS systems.
- Natural speech samples.
- Large-scale subjective evaluations (Mean Opinion Scores - MOS) from hundreds of listeners.
They specifically used English tasks from these years, excluding reportorial and paragraph domains. The data was normalized per year and per system to mitigate corpus-specific acoustic differences.
Key Insights and Ideas
- Hierarchical prediction is powerful: Combining system-level and stimulus-level predictions, especially by predicting their difference, significantly improves naturalness prediction accuracy compared to predicting stimulus-level scores alone. This suggests that understanding the overall system's performance provides valuable context for evaluating individual utterances.
- CNNs capture local and global features: CNNs show promise in overcoming the limitations of traditional frame-wise analysis by automatically learning features that can capture both short-term spectral discontinuities and longer-term prosodic patterns.
- NNs outperform linear models with large data: The availability of large-scale subjective data from the Blizzard Challenge allows neural networks to significantly outperform linear regression methods, especially when dealing with complex patterns.
- Subjective scores are predictable: Bootstrap experiments confirm that human listener ratings of synthetic speech naturalness are indeed predictable, with good agreement among listeners.
Key Results
- The hierarchical approach, particularly using NN+NN (both system and stimulus predictors are NNs), achieved the lowest system-level RMSE, indicating strong performance in predicting overall synthesizer quality.
- The hierarchical structure improved stimulus-level prediction accuracy, showing higher correlations with observed MOS.
- CNNs, when integrated into the hierarchical framework (LR+CNN), showed improved within-system stimulus-level correlations, suggesting they are better at identifying local signal features that affect human perception within a specific system.
- While CNNs did not always outperform traditional NNs on overall test-set error, they demonstrated success in identifying local signal features affecting human perceptual response.
Limitations and Future Work
- Limited training data for NN+LR: The authors excluded the NN+LR combination due to the relatively small number of TTS systems available in the dataset (85 systems), which might not be sufficient for training such a model effectively.
- CNN performance plateau: Despite improvements in training error, the test-set error for CNNs did not significantly improve. The authors attribute this to acoustic variations from different linguistic contexts, speakers, and speaking styles, which are difficult to capture with the current feature extraction and limited data.
- Future Work:
- Augmenting synthetic-speech acoustic features with linguistic information (e.g., phoneme identity, prosodic context).
- Investigating the utility of these objective naturalness predictions for improving TTS system development directly.
Summary updated