TL;DR
This paper proposes a novel approach to multilingual speech emotion recognition (mSER) by modeling the human perception of emotion in three layers: acoustic features, semantic primitives, and emotion dimensions. They introduce new wavelet transform-based acoustic features that significantly outperform traditional methods in capturing subtle emotional cues across different languages.
What Problem Does This Paper Address?
Recognizing emotions in speech is crucial for natural human-computer interaction, especially in multilingual contexts where language diversity poses a significant challenge. Existing speech emotion recognition (SER) systems often struggle to generalize across languages due to limitations in acoustic feature extraction and the complexity of human emotion perception. While many models focus on extracting features like Mel-frequency cepstral coefficients (MFCCs) or prosodic features, these often fail to capture the dynamic, localized emotional information or generalize well across diverse speakers and languages. Furthermore, many deep learning approaches act as "black boxes," making it difficult to understand how emotion is perceived. This paper aims to bridge this gap by developing a computational model that mimics the multi-layered perceptual process of emotion, focusing on language-independent acoustic features.
Methods
The core of this paper lies in its novel approach to acoustic feature extraction and its three-layer emotion perception model.
- Wavelet Transform (WT) for Acoustic Features: Instead of relying solely on traditional Fourier Transform-based methods (like STFT for MFCCs), the authors employ the Discrete Wavelet Transform (DWT) on various low-dimensional features (LDFs) extracted from glottal source and vocal tract signals. This allows for a multi-resolution analysis, decomposing the signal into different frequency bands and time scales. This is particularly beneficial for capturing localized, dynamic emotional events that might be missed by fixed-resolution methods. They extract wavelet energy and entropy-related features from these decomposed signals.
- Three-Layer Emotion Perception Model: This model is inspired by Brunswik's lens principles and Scherer's work.
- Acoustic Features Layer: This layer uses the newly proposed WT-based acoustic features.
- Semantic Primitives Layer: This layer bridges the gap between raw acoustic cues and abstract emotional concepts. The authors use 17 semantic primitives (e.g., "bright," "dark," "strong," "weak") that were previously identified as relevant for describing emotional speech.
- Emotion Dimensions Layer: This final layer maps the processed information onto valence (pleasantness/unpleasantness) and arousal (relaxation/excitement) dimensions, which are widely accepted as fundamental ways to describe emotional states.
- Adaptive Neuro-Fuzzy Inference Systems (ANFIS): The model utilizes ANFIS for its ability to fuse human knowledge (from semantic primitive evaluations) with nonlinear processing, making it suitable for the fuzzy nature of emotion perception.
- Sequential Floating Forward Selection (SFFS): This algorithm is used to select the most informative subset of acoustic features and semantic primitives, optimizing the model's performance and reducing computational cost.
Datasets
The study uses three publicly available corpora of acted emotions:
- Fujitsu Database (Japanese): Features acted emotions from one professional actress.
- Berlin Emo-DB (German): Features acted emotions from ten professional actors.
- CASIA Corpus (Chinese): Features spontaneous content from four actors.
For training and evaluation, the authors selected four common emotions across these corpora: neutral, happy, angry, and sad.
Key Insights and Ideas
- Wavelet Transform's Superiority: The WT-based features are shown to be more effective than traditional prosodic and spectral features because they can better capture the dynamic, localized, and multi-resolution nature of emotional speech signals. This is crucial for distinguishing subtle emotional nuances.
- Multi-Layered Perception is Key: The paper reinforces the idea that human emotion perception is not a direct mapping from acoustic cues to emotion dimensions. Instead, it involves intermediate steps like processing semantic primitives, which helps in generalizing across languages and speakers.
- Language-Independent Cues: The WT-based features are hypothesized to capture more universal, language-independent emotional cues, which is vital for multilingual SER.
- Bridging Acoustic and Semantic: The model effectively bridges the gap between low-level acoustic features and high-level semantic and dimensional representations of emotion.
Key Results
The proposed WT-based acoustic features achieved the best performance in estimating both valence and arousal dimensions across the multilingual corpora.
- Valence Estimation: Achieved a correlation coefficient (CC) of 0.82 and a mean absolute error (MAE) of 0.47. This represents a significant improvement over baselines, with relative error reductions of up to 58% for CC and 35.6% for MAE.
- Arousal Estimation: Achieved a CC of 0.93 and an MAE of 0.30. This also outperformed baselines, showing relative error reductions of up to 46% for MAE.
These results significantly outperformed traditional prosodic features (PSFs), statistical modulation spectral features (MSFs), and their combination.
Limitations and Future Work
- Acted Emotions: The study primarily uses acted emotions. While this is common in SER research, it may not fully represent the nuances of spontaneous emotional speech. The authors mention using spontaneous content from the CASIA corpus, but the primary focus seems to be on acted emotions for feature evaluation.
- Feature Selection: While SFFS is used to select optimal features, the paper doesn't extensively discuss the interpretability of the specific 21 selected WT-based features beyond their general categories (energy, entropy).
- Future Work: The authors suggest that their approach is promising for modeling the dynamic process of emotional speech across languages and could be integrated into affective speech-to-speech translation systems. Further research could explore more spontaneous emotion datasets and delve deeper into the interpretability of the selected features.
Summary updated