TL;DR
This paper investigates which short-term speech features (spectral and excitation) are most crucial for generating emotionally expressive synthesized speech using statistical parametric methods. They found that spectral features, specifically mel-LSPs, are more important than excitation features for conveying emotion in synthesized speech.
What Problem Does This Paper Address?
Generating emotionally expressive speech is a key goal in speech synthesis. While it's generally understood that prosody plays a significant role in emotion, the specific contribution of short-term spectral and excitation parameters within the context of Hidden Markov Model (HMM)-based statistical parametric speech synthesis (SPSS) remains less clear. This paper aims to dissect this by analyzing how different parameterizations of these short-term features correlate with emotional states and impact the perceived expressiveness of synthesized speech.
Methods
The authors employ a two-pronged approach:
- Emotion Classification: They first assess how well different speech parameters can predict the emotion of a speech segment. This is done using two methods:
- K-means Clustering: This unsupervised method groups feature vectors into clusters, and the performance is measured by how well these clusters align with known emotional labels.
- Gaussian Mixture Model (GMM)-based Emotion Identification: This supervised method trains GMMs for each emotion and then uses them to classify unseen data.
- Synthesizer Training and Evaluation: Based on the classification results, the most informative parameters are selected to train emotion-dependent HMM synthesizers. The quality and emotional similarity of the synthesized speech are then evaluated through listening tests.
The specific parameters analyzed include:
- Spectral Parameters: Mel-cepstrum (MCEP) and Mel-Line Spectral Pairs (mel-LSP).
- Excitation Parameters:
- Complex cepstrum (ACC) for glottal pulse modeling.
- Time-domain smooth group delay (TSGD) for glottal pulse modeling.
- Band-aperiodicity coefficients (BAP) for controlling the mix of pulse and noise.
Datasets
The study utilized a North-American English emotional database spoken by a female speaker. The database contained 609 sentences for each of the four emotion classes: neutral, angry, happy, and sad. This database was segmented at the phone level.
Key Insights and Ideas
- Spectral Dominance: Mel-LSP parameters emerged as the most effective spectral features for identifying emotion, suggesting that the shape of the vocal tract's resonance is a strong indicator of emotional state.
- Excitation Ambiguity: The study found less clear evidence for the superiority of any single excitation parameterization in predicting emotion. This suggests that while excitation is important, its emotional encoding might be more complex or less directly captured by the analyzed features.
- Parameter Interaction: The listening tests indicated that using the "matching" emotional spectral models significantly improved the similarity of synthesized speech to natural speech, whereas the impact of matching excitation streams was less pronounced. This highlights the primary role of spectral characteristics in conveying emotion in this synthesis framework.
Key Results
- GMM Classification: Mel-LSP parameters achieved the highest emotion identification accuracy among spectral features. For excitation features, the results were less conclusive.
- Listening Tests: When synthesizing speech, using the correct emotional spectral models (mel-LSP) led to higher similarity scores compared to using mismatched spectral models. The impact of matching excitation parameters (complex cepstrum, band-aperiodicity) on similarity was not as significant.
Limitations and Future Work
The authors acknowledge several limitations:
- Database Size: The use of a relatively small database might have limited the ability to fully capture the nuances of emotional expression.
- Excitation Feature Representation: The complex cepstrum and group delay features might not fully capture the intricate details of glottal excitation, and issues like pitch marking errors and phase unwrapping inaccuracies could have affected results.
Future work will focus on addressing these limitations by:
- Using larger and more diverse emotional databases.
- Exploring more advanced methods for representing glottal pulse features to better capture short-term glottal information.
- Providing a more accurate analysis of glottal pulse features and their connection to emotion in HMM-based synthesizers.
Summary updated