TL;DR
This paper tackles the challenge of making Text-to-Speech (TTS) systems more emotionally expressive by introducing a novel "emotion intensity" input. They propose two unsupervised methods to extract this intensity from audio, allowing TTS models to better convey nuanced emotions without requiring explicit emotion labels for every word. Their findings show that this intensity input significantly improves the perceived emotional expressiveness of synthetic speech.
What Problem Does This Paper Address?
Current Text-to-Speech (TTS) systems, while excellent at producing natural-sounding neutral speech, struggle with conveying a wide range of emotions. Creating emotional speech datasets is a significant hurdle due to the cost and time involved in recording and labeling diverse emotional expressions with varying intensities. Emotions aren't static; they ebb and flow within an utterance. Traditional TTS models often treat emotions as a single, global label for an entire sentence, failing to capture these temporal dynamics. This paper aims to bridge this gap by enabling TTS systems to leverage fine-grained, time-varying emotion intensity information extracted automatically from audio.
Methods
The core of this work lies in developing methods to extract "emotion intensity" from audio without explicit frame-level emotion labels. They propose two main approaches:
- Attention LSTM: They utilize a pre-trained emotion recognition model based on a Bidirectional LSTM with an attention mechanism. The intuition here is that the attention weights, which highlight important parts of the audio for emotion classification, can serve as a proxy for emotion intensity over time.
- Transformer with Saliency Maps: For more complex, state-of-the-art transformer-based emotion recognizers, they employ saliency maps. Saliency maps are techniques used to understand which parts of the input (in this case, audio features) are most influential for the model's prediction. They explore various saliency map methods (Input Gradients, Smoothgrad, Input X Gradient, Integrated Gradients) and aggregation strategies (max, mean) to derive an intensity signal.
Additionally, they compare their methods against a "Attribute Rank" approach, which learns a ranking function to assign emotion intensity levels based on relative attributes.
Datasets
The researchers used two primary datasets:
- IEMOCAP: This dataset was used for pre-training the emotion recognizer models. It contains acted and spontaneous emotional speech from multiple speakers.
- SAVEE: This is the primary dataset used for training and evaluating the emotional TTS system. It's an audio-visual British English database with phonetically balanced sentences spoken with seven emotions. The authors note that SAVEE is relatively small for training complex TTS models, necessitating pre-training on IEMOCAP and WSJCAMO.
Key Insights and Ideas
- Attention as Emotion Intensity: The attention weights from a well-trained emotion recognition LSTM can effectively capture and represent the temporal dynamics of emotion intensity, proving to be a surprisingly effective and simple method.
- Saliency Maps for Complex Models: While transformer models offer better emotion recognition accuracy, extracting a meaningful emotion intensity signal from them using saliency maps is more challenging and requires careful selection of techniques.
- Unsupervised Extraction is Key: The ability to extract emotion intensity without explicit frame-level labels is crucial for scaling emotional TTS, as such labels are scarce and expensive to obtain.
- Intensity Matters for Expressiveness: Providing the TTS model with this fine-grained intensity input, even when extracted from reference audio, significantly boosts the perceived emotional expressiveness of the synthesized speech.
- Scaling Intensity: The TTS models learned to associate the intensity input with speech characteristics, allowing for natural scaling of emotion intensity (e.g., making speech more energetic for happy or angry utterances).
Key Results
- Subjective Evaluation: In listening tests, the TTS system using attention weights from the attention LSTM as emotion intensity achieved the highest accuracy in terms of perceived emotion compared to baseline, transformer-based saliency maps, and attribute rank methods.
- No Audio Quality Degradation: The proposed emotion intensity inputs did not negatively impact the overall audio quality of the synthesized speech, as confirmed by MOS tests.
- Transformer vs. Attention: While the transformer model achieved higher emotion recognition accuracy, the simpler attention LSTM method provided a more effective emotion intensity signal for TTS. This suggests that the best emotion recognition model doesn't necessarily yield the best intensity signal for synthesis.
Limitations and Future Work
- Saliency Map Challenges: The authors acknowledge that saliency map techniques, often developed for vision tasks, may not be optimal for speech and might require adaptation or more speech-specific convolutional networks.
- Predicting Intensity from Text: A significant limitation is the lack of a method to predict emotion intensity directly from text. Future work will focus on developing such prediction capabilities.
- Conversion Applications: The authors suggest exploring the use of this technique for emotion intensity conversion in speech-to-speech translation.
- Attribute Rank Data Needs: The attribute rank method, while performing worse in their experiments, requires very little training data, making it a potential candidate for low-resource scenarios.
Summary updated