TL;DR
This paper introduces a novel set of speech features called Amplitude Modulation Cepstral Coefficients (AMCC) designed to improve the accuracy of automatic speech emotion recognition. By focusing on the amplitude modulation of speech signals, AMCC features demonstrate a significant improvement over traditional spectral features like MFCCs, offering a more robust way to capture emotional nuances in voice.
What Problem Does This Paper Address?
Recognizing human emotions from speech (Speech Emotion Recognition or SER) is a challenging but important task with applications ranging from call centers to human-robot interaction. A key hurdle in SER is extracting speech features that effectively capture emotional states while being robust to variations in speaker identity and the actual content of the speech. Traditional spectral features like Mel-Frequency Cepstral Coefficients (MFCCs) are widely used but can be sensitive to noise and frequency perturbations, potentially limiting their effectiveness in capturing subtle emotional cues. This paper aims to address this limitation by proposing a new feature extraction method that leverages amplitude modulation information, which is hypothesized to be more directly related to emotional expression.
Methods
The core contribution of this paper is the development of Amplitude Modulation Cepstral Coefficients (AMCC). The AMCC feature extraction process begins by decomposing the speech signal into multiple frequency bands using a gammatone filterbank. For each band, the amplitude modulation (AM) power spectrum is estimated using a Smoothed Nonlinear Energy Operator (SNEO). This SNEO is an advancement over the standard nonlinear energy operator (NEO) and is designed to be more robust. The AM power spectrum is then processed through a power-law nonlinearity (with a coefficient of 0.07, found to be more robust than logarithmic nonlinearity) and finally transformed using a Discrete Cosine Transform (DCT). To enhance the temporal dynamics, delta and double delta features are computed and appended. For comparison, the paper also uses standard MFCCs and Mel-warped DFT spectrum-based cepstral coefficients (MWDCC). The extracted features are then modeled using Gaussian Mixture Models (GMMs) for emotion classification.
Datasets
The primary dataset used for training and evaluation in this paper is the FAU AIBO spontaneous emotion corpus. This corpus consists of spontaneous recordings of German children interacting with a pet robot. It contains approximately 9959 chunks for training and 8257 chunks for testing. The chunks are labeled into five emotion categories: Anger (A), Emphatic (E), Neutral (N), Positive (P), and Rest (R). The paper notes that this dataset exhibits a highly unbalanced class distribution.
Key Insights and Ideas
- Amplitude Modulation as a Richer Source of Emotional Information: The paper posits that the way amplitude changes over time within different frequency bands of speech is a more direct indicator of emotional state than spectral characteristics alone.
- SNEO for Robust AM Estimation: The use of a Smoothed Nonlinear Energy Operator (SNEO) is crucial for reliably estimating the amplitude modulation components, making the features more resilient to signal variations.
- Power-Law Non-linearity for Improved Robustness: Employing a power-law non-linearity, rather than the traditional logarithmic one, is shown to yield more robust features, particularly in the presence of signal perturbations.
- AMCC Outperforms Traditional Spectral Features: The experimental results strongly suggest that AMCC features provide a significant improvement in emotion recognition accuracy compared to established spectral features like MFCC and MWDCC.
Key Results
The experimental evaluation on the FAU AIBO corpus shows that the proposed AMCC features outperform the baseline MFCC and MWDCC systems.
- In terms of Unweighted Average Recall (UAR), AMCC achieved 44.50%, compared to MFCC at 43.09% and MWDCC at 43.11%. This represents a relative improvement of approximately 3.2% over both baseline systems.
- In terms of Weighted Average Recall (WAR), AMCC achieved 42.58%, compared to MFCC at 37.99% and MWDCC at 41.48%. This shows a more substantial relative improvement of approximately 12.0% over MFCC and 2.65% over MWDCC.
Limitations and Future Work
The authors acknowledge that the class distribution in the FAU AIBO corpus is highly unbalanced, which can affect performance metrics. They suggest future work to:
- Compare the performance of AMCC with other advanced modulation-based features like T-MFCC and TEMFCC.
- Explore the fusion of spectral and prosodic features with the proposed AMCC features to further enhance emotion recognition accuracy.
Summary updated