TL;DR
This paper tackles the challenge of recognizing emotions in speech under noisy, real-world conditions, where traditional methods falter. By analyzing the robustness of acoustic features to noise and reverberation, they propose a feature selection strategy that significantly improves affect recognition performance, especially when combined with multi-condition training.
What Problem Does This Paper Address?
Automatic emotion recognition from speech has made strides in controlled laboratory settings. However, its performance degrades significantly in real-life scenarios due to factors like background noise, reverberation, and variations in affective expression. This paper focuses specifically on the impact of degraded acoustic conditions (noise and reverberation) on affect recognition and proposes a method to select acoustic features that are more resilient to these challenges, thereby improving the generalizability of recognition systems.
Methods
The core of this paper lies in its feature selection strategy. The authors first extract a comprehensive set of acoustic features (6373 features from the ComParE feature set) from speech signals. They then employ a Correlation Coefficient based Feature Selection (CC-FS) method. This involves calculating the Pearson correlation coefficient between each feature and the target emotion labels (valence, arousal, and interest) across various acoustic conditions. Features that maintain a strong correlation across a range of noisy and reverberated environments are identified as robust.
To evaluate their approach, they utilize Multi-Condition Training (MCT). This involves training the classifier on data that has been artificially degraded with different types and levels of noise and reverberation. They then compare the performance of classifiers trained on clean data versus those trained on these multi-condition datasets, with and without their proposed feature selection. Support-Vector Machines (SVMs) with linear kernels are used as the classification model.
Datasets
The study uses two primary datasets:
- GEMEP (Geneva Multimodal Emotion Portrayals): This corpus contains acted emotional speech with continuous dimensional observer annotations for valence and arousal. For this study, a subset with 154 instances and observer annotations was used.
- TUM Audio-Visual Interest Corpus (TUM-AVIC): This corpus captures natural, spontaneous affect and was used for the interest sub-challenge of the INTERSPEECH 2010 Paralinguistic Challenge. It provides continuous labels for the level of interest.
To simulate real-life conditions, both datasets were artificially degraded by convolving them with Room Impulse Responses (RIRs) from the Aachen Impulse Response Database and mixing them with additive noise samples (babble, city street, and music). This resulted in multiple test and training sets representing various combinations of reverberation and Signal-to-Noise Ratios (SNRs).
Key Insights and Ideas
- Noise and Reverberation Significantly Degrade Performance: The paper clearly demonstrates that real-world acoustic conditions severely impact affect recognition accuracy, with performance drops of up to 10% for reverberation and an additional 8% for additive noise at 0 dB SNR.
- Feature Robustness Varies Greatly: Not all acoustic features are equally affected by noise. Prosodic features, like energy-related ones, tend to become less reliable in noisy conditions, while temporal features might become more important as they capture more stable signal distribution over time.
- Feature Selection is Crucial for Generalization: Selecting features that are robust across multiple noisy conditions (using CC-FS on multi-condition training data) significantly boosts classifier performance, even when training on clean data. This suggests that features important in clean conditions might not generalize well to noisy ones.
- Multi-Condition Training and Feature Selection are Synergistic: The combination of training on diverse noisy conditions (MCT) and selecting features robust to these conditions (CC-FS) yields the most significant performance gains, leading to more robust affect recognition systems.
- Valence is Inherently Difficult from Acoustics Alone: The paper notes that even under ideal acoustic conditions, achieving high accuracy for valence recognition from speech alone is challenging, highlighting the limitations of acoustic cues for this specific emotion dimension.
Key Results
- The proposed feature selection method (CC-FS) combined with Multi-Condition Training (MCT) significantly improves Unweighted Average Recall (UAR) across various acoustic conditions.
- For arousal and valence, using the top 400 features selected via CC-FS on multi-condition data outperformed using the full feature set, especially under challenging acoustic conditions.
- For LOI determination, MCT showed a substantial improvement, and feature selection further contributed to robustness, with the worst MCT result (at 0 dB SNR) being comparable to the best result without MCT.
- The study shows that feature selection can achieve similar or better results with a fraction of the original features (e.g., 6.3% of the ComParE set), leading to more efficient models.
Limitations and Future Work
- Limitations: The paper acknowledges that the performance for interest recognition is lower than in some previous studies, partly due to using a more comprehensive set of instances and normalizing utterances to peak amplitude, which removes mean energy as a cue. The inherent difficulty of acoustic valence recognition is also noted.
- Future Work: The authors suggest a more detailed study of individual feature relevance for specific SNR conditions to understand noise robustness better. They also propose combining human perception tests with automatic analysis to correlate human and machine performance degradation and to guide the development of more human-like acoustic affect perception systems for real-world applications.
Summary updated