TL;DR
This paper explores using glottal source modeling (GlottHMM) to improve expressive speech synthesis, moving beyond traditional prosodic features. They demonstrate that glottal features significantly boost recognition rates for both speaking styles and emotions, and show that these features are robust to speaker bias when properly normalized.
What Problem Does This Paper Address?
Creating truly human-like synthetic speech requires more than just mimicking the rhythm and pitch of human speech. Expressive speech synthesis aims to imbue synthetic voices with emotional and stylistic nuances, making them more adaptable and natural. The challenge lies in capturing these subtle expressivity cues, which are not fully represented by traditional prosodic features like fundamental frequency (F0) and speaking rhythm. This paper investigates whether modeling the glottal source, the very origin of voice production, can provide the missing layer of information needed for more controllable and expressive speech synthesis.
Methods
The core of this research is the use of GlottHMM, a vocoding technique that decomposes speech into its glottal source and vocal tract components. GlottHMM models the vocal tract using Line Spectral Frequencies (LSFs) and the glottal source with its spectral tilt, also using LSFs. Crucially, it also extracts features like F0, harmonics-to-noise ratio (HNR), and introduces novel features such as Normalized Amplitude Quotient (NAQ) and magnitude differences between the first 10 harmonics.
The authors employ several analytical techniques:
- InfoGain analysis: To determine the individual relevance of different features for classification.
- Speaker Normalization: Applying z-score normalization to individual speaker data to mitigate speaker-specific biases.
- Kullback-Leibler (KL) distance: To measure the dissimilarity between feature distributions.
- Multidimensional Scaling (MDS): To visualize the relationships between different emotional or stylistic states in a lower-dimensional space.
Datasets
The study utilizes four distinct databases:
- Emotional Speech Databases:
- SES (Spanish Emotional Speech): Acted emotional speech.
-
SEV (Spanish Expressive Voices): Acted emotional speech. Both SES and SEV feature a subset of basic emotions (happiness, anger, sadness, surprise, neutral, and in SEV, cold anger, fear, disgust, hot anger, panic/frightened).
-
Speaking Style Databases:
- Ircam: A French multi-speaker, multi-style database with four formal styles (news, live sports, political speeches, mass sermons).
- C-oral-ROM: A large, multi-language, multi-style database with a wide range of formal and informal speaking styles extracted from media and conversations. For this study, only Spanish formal media styles were analyzed.
Key Insights and Ideas
- Glottal Source is Key for Expressivity: The research strongly suggests that glottal source features carry significant expressivity information that is not captured by prosodic features alone.
- Beyond F0 and Rhythm: While F0 and rhythm are important, certain glottal parameters (like specific LSF means/variances and HNR values) proved to be even more informative for distinguishing speaking styles.
- Robustness to Speaker Bias: After applying speaker normalization, glottal features demonstrate consistency across different speakers, suggesting they are not overly sensitive to individual vocal characteristics. This is crucial for building generalizable expressive speech systems.
- Visualizing Expressive Spaces: MDS plots effectively visualize how different emotions and speaking styles cluster, providing perceptual insights into the separability of these states based on the analyzed features.
Key Results
- Style Recognition: Using GlottHMM features on the Ircam database achieved a remarkable 95.4% recognition rate for speaking styles, a significant improvement over purely prosodic methods (around 74%).
- Emotion Recognition: For emotional speech (SES and SEV), the system achieved 81.2% recognition rate with glottal features.
- Feature Importance: InfoGain analysis revealed that features like LSF means and variances, and HNR values, were often more informative than F0 or rhythm for style detection.
- Spanish Style Separation: A proposed speaking style distribution for Spanish media, based on F0 and speaking rate, showed distinct clusters for styles like news, sports commentary, and political speech, correlating with environmental control and speech spontaneity.
Limitations and Future Work
- Database Limitations: The authors note that current style databases, like C-oral-ROM, often suffer from variability in recording environments and a high number of speakers with limited data, making consistent style representation challenging.
- Data Labeling: While raw data for style analysis is abundant, adequate labeling for training robust models remains a hurdle.
- Future Directions: The paper suggests that controlling recording environments and speaker variability during database creation could lead to more consistent and reliable expressive speech analysis. Further research into normalizing for average neutral voices is also proposed to aid recognition.
Summary updated