TL;DR
This paper introduces EmoAug, an unsupervised speaking style transfer model that enriches emotional speech data by altering prosodic attributes like stress and rhythm while preserving semantic content and speaker identity. By generating diverse emotional speech samples, EmoAug effectively addresses data scarcity and class imbalance in Speech Emotion Recognition (SER), leading to state-of-the-art performance.
What Problem Does This Paper Address?
Speech Emotion Recognition (SER) is crucial for enhancing human-machine interaction, but its progress is hampered by a lack of large-scale, naturally emotional speech datasets. Existing datasets are often collected under simulated conditions, making them less representative of real-world emotional expression. Furthermore, the inherent variability in how individuals express emotions, coupled with the tendency for certain emotions (like "Neutral") to be overrepresented, leads to data imbalance and model overfitting. While data augmentation techniques exist, many borrowed from Automatic Speech Recognition (ASR) can inadvertently alter emotional expression by changing pitch or speed. Other methods that manipulate emotional features can struggle with intuitive evaluation and may create conflicts between prosody and content.
Methods
EmoAug tackles these challenges with a novel unsupervised approach. It comprises three main components:
- Speech Quantization and Semantic Encoder: Instead of relying on traditional ASR models (which require extensive labeled data and can be sensitive to noise), EmoAug uses HuBERT representations to capture semantic speech content. These representations are then quantized into discrete cluster labels using k-means. A semantic encoder, consisting of Conv1D layers and an LSTM, processes these quantized labels to extract semantic information.
- Paralinguistic Encoder: This component, based on the ECAPA-TDNN model, is designed to learn non-verbal information such as speaking styles, emotion states, and speaker identities from the input audio. Crucially, it's initialized with a pre-trained model and employs techniques like attentive statistics pooling to prevent semantic information from leaking into the paralinguistic representation.
- Decoder: A Tacotron2-based decoder with location-aware attention reconstructs speech signals. It takes input from both the semantic and paralinguistic encoders to generate mel-spectrograms. For style transfer, the paralinguistic encoder's input is replaced with a reference speaking style.
- Discriminator: During fine-tuning, a discriminator is introduced to specifically differentiate pitch changes between genuine and synthesized speech, aiming to improve the quality of the generated audio.
The entire process is unsupervised, meaning it doesn't require paired speech or emotion labels for training.
Datasets
- LRS3-TED: Used for pre-training EmoAug. This large-scale dataset, containing over 400 hours of spontaneous speech from TED and TEDx talks, provides diverse speaking styles and emotions, aiding the model in learning rich paralinguistic variations.
- IEMOCAP: Used for fine-tuning and evaluation. This multimodal emotion dataset, recorded with 10 actors in a fictitious scenario, contains four emotions (happy, sad, angry, neutral) and is used with a leave-one-session-out cross-validation strategy.
Key Insights and Ideas
- Unsupervised Style Transfer for SER: The core innovation is using unsupervised speaking style transfer to generate diverse emotional speech, directly addressing the data scarcity problem in SER without needing paired data.
- Decoupling Semantics and Prosody: By using HuBERT for semantic content and a separate paralinguistic encoder for style, the model effectively separates what is said from how it is said, allowing for targeted manipulation of prosody.
- Quantized Semantic Representation: Moving away from traditional ASR models to quantized HuBERT representations makes the system more robust to noise and less reliant on text transcriptions, which are often unavailable or imperfect for emotional speech.
- Discriminator for Pitch Quality: The introduction of a pitch-specific discriminator during fine-tuning is a clever way to enhance the perceptual quality of the synthesized speech without overly complicating the main generation process.
- Data Augmentation for Imbalance: EmoAug not only generates new samples but also allows for controlled generation of samples for under-represented classes, directly tackling the data imbalance issue.
Key Results
- State-of-the-Art Performance: SER models trained with EmoAug-generated data significantly outperform previous supervised and self-supervised methods on the IEMOCAP dataset, achieving a Weighted Accuracy (WA) of 72.66% and Unweighted Accuracy (UA) of 73.75%.
- Improved Representation Separation: Visualizations (t-SNE) show that models trained with EmoAug data exhibit clearer separation of emotion representations compared to models trained with only original data.
- Effectiveness of Augmentation: Increasing the augmentation time (number of style transfers per utterance) consistently improved SER performance, with the discriminator further enhancing these gains.
- Preservation of Identity and Content: Qualitative assessments and experimental results indicate that EmoAug successfully transfers speaking styles while retaining speaker identity and semantic content.
Limitations and Future Work
- Data Scarcity Remains a Challenge: While EmoAug alleviates data scarcity, the fundamental challenge of obtaining truly natural and spontaneous emotional speech datasets persists.
- Potential for Overfitting: Despite efforts like early stopping and scheduled sampling, the authors acknowledge the potential for overfitting, especially with limited fine-tuning data.
- Future Work: The authors suggest exploring more sophisticated methods for semantic representation and paralinguistic feature extraction. They also propose investigating the model's performance on datasets with more diverse emotions and acoustic conditions. The ability to control specific prosodic attributes (e.g., intensity of stress) could also be a future research direction.
Summary updated