TL;DR
This paper presents a novel approach to synthesizing emotional speech using Hidden Markov Models (HMMs) without requiring emotional speech data from the target speaker. By interpolating a target speaker's neutral speech model with an emotional model from a pool of speakers, they achieve expressive synthesis while preserving speaker identity.
What Problem Does This Paper Address?
Creating realistic emotional speech synthesis is challenging. Traditional methods often rely on large amounts of emotional speech data from the target speaker, which can be difficult and expensive to collect. This paper tackles the problem of synthesizing emotional speech for a specific target speaker without needing that speaker to provide any emotional recordings. The goal is to leverage existing emotional speech data from other speakers to imbue a target speaker's neutral voice with emotion, while maintaining their unique vocal characteristics.
Methods
The core of the proposed method is model interpolation. Instead of directly training an emotional model for the target speaker, they train a neutral HMM for the target speaker and then interpolate it with an emotional HMM from a pre-existing pool.
Here's a breakdown of their approach:
- HMM-Based Speech Synthesis: The foundation is a standard HMM-based speech synthesis system, which involves training models on speech data to capture spectral and fundamental frequency (F0) characteristics.
- Model Pool: They create a pool of pre-trained HMMs. This pool contains both neutral and emotional models from various speakers.
- Target Speaker Neutral Model (NT): They train a neutral HMM specifically for the target speaker using their neutral speech data.
- Emotional Model Selection: To choose which emotional model to interpolate with, they employ a Monophone-based Mahalanobis Distance (MBMD). This metric measures the difference between two HMMs at the monophone (single sound) level, considering both mean and covariance of feature distributions.
- They first find a speaker X whose neutral model (Nx) is closest to the target speaker's neutral model (NT) using MBMD. This helps in selecting a speaker with similar vocal characteristics.
- Then, they find an emotional model (Ez) from the pool that is closest to the target speaker's neutral model (NT) using MBMD. This step aims to find an emotional model that, when adapted, might sound natural on the target speaker.
- Interpolation Ratio Determination: The crucial step is determining the interpolation ratio (R) between the target speaker's neutral model (NT) and the selected emotional model (Ez). They achieve this by finding the ratio R that makes the interpolated model
I(NT, Ez, R)closest to the emotional model (Ex) of the speaker X identified earlier. This ensures the synthesized speech is emotionally expressive but also resembles the target speaker's vocal identity. - Final Interpolation: The final emotional model (ET) is generated by interpolating NT and Ez with the determined ratio R. This interpolated model is then used for synthesis.
The MBMD is a novel contribution for model selection and interpolation weight determination in this context.
Datasets
- Mandarin Speech Database: The authors collected a Mandarin speech database specifically for emotional speech synthesis.
- Content: It includes three target emotions: "angry," "sad," and "happy."
- Speakers: Three male speakers recorded each emotion and neutral speech.
- Data Split: Each speaker recorded 900 neutral utterances and 900 emotional utterances (300 for each emotion).
- Model Training: Neutral models were trained on 900 utterances, and emotional models on 300 utterances.
- Target Speaker Data: For evaluation, 100 neutral utterances were used to train the target speaker's neutral model, and 30 utterances per emotion were used for subjective tests.
Key Insights and Ideas
- Emotional Synthesis Without Target Emotion Data: The most significant insight is that you can achieve emotional speech synthesis for a target speaker without needing them to record any emotional speech. This dramatically reduces data collection costs and complexity.
- Speaker Identity Preservation: By carefully selecting an emotional model and determining the interpolation ratio based on speaker similarity (using MBMD), the system successfully preserves the target speaker's vocal identity. This is a critical factor for natural-sounding synthesis.
- MBMD as a Robust Metric: The Monophone-based Mahalanobis Distance proves to be an effective metric for comparing HMMs, enabling intelligent model selection and interpolation weight determination. It's more robust than simpler distance measures when dealing with complex HMM structures.
- Interpolation as a Flexible Control Mechanism: Model interpolation offers a flexible way to blend characteristics from different models, allowing for fine-grained control over the synthesized speech's emotional expressiveness and speaker identity.
Key Results
Subjective evaluation tests demonstrated the effectiveness of their approach:
- Emotional Expressiveness (ABX Test): The synthesized speech achieved accuracy rates of 77% for Angry, 73% for Happy, and 77% for Sad in distinguishing the synthesized emotional speech from reference emotional speech. This indicates high emotional expressiveness.
- Naturalness (MOS Test): The Mean Opinion Scores (MOS) for naturalness were comparable to synthesized neutral speech, with scores around 3.3-3.7 (on a 5-point scale). This suggests the emotional synthesis did not degrade the naturalness of the speech.
- Speaker Identity (Similarity Test): The similarity test scores (MOS) for speaker identity were around 3.1-3.4 (on a 5-point scale), indicating that the synthesized speech sounded like the target speaker, better than "slightly similar" on average.
Limitations and Future Work
- Limited Emotions and Speakers: The current work focuses on three specific emotions and a small pool of male speakers. Expanding to a wider range of emotions and speaker demographics (e.g., female speakers) would be beneficial.
- MBMD Complexity: While effective, MBMD involves calculating covariance matrices, which can be computationally intensive. Exploring more efficient distance metrics might be a future direction.
- Interpolation Ratio Grid Search: The paper mentions using a grid search for the interpolation ratio. More sophisticated optimization techniques could potentially yield better results or faster convergence.
- Further Exploration of Interpolation: The authors suggest that the interpolation scheme can be applied to spectral, F0, and state-duration models, implying potential for further refinement by optimizing each component independently or jointly.
Summary updated