Abstract
Speech emotion recognition (SER) often experiences reduced performance due to background noise. In addition, making a prediction on signals with only background noise could undermine user trust in the system. In this study, we propose a Noise Robust Speech Emotion Recognition system, NRSER. NRSER employs speech enhancement (SE) to effectively reduce the noise in input signals. Then, the signal-to-noise-ratio (SNR)-level detection structure and waveform reconstitution strategy are introduced to reduce the negative impact of SE on speech signals with no or little background noise. Our experimental results show that NRSER can effectively improve the noise robustness of the SER system, including preventing the system from making emotion recognition on signals consisting solely of background noise. Moreover, the proposed SNR-level detection structure can be used individually for tasks such as data selection.
Introduction
Speech Emotion Recognition (SER) is a critical component for empathetic AI, but it faces two major hurdles in real-world applications. First, background noise significantly degrades the accuracy of emotion detection. Second, while Speech Enhancement (SE) algorithms can clean up audio, they often introduce "processing artifacts" that can degrade performance when applied to already clean signals. Furthermore, systems that attempt to predict emotions from pure background noise (e.g., a passing car) can undermine user trust. The proposed NRSER system addresses these issues by adaptively applying speech enhancement based on the estimated noise level of the input.
The NRSER Framework
The NRSER system is designed to balance noise reduction with signal preservation through three primary neural network blocks and a novel reconstitution strategy.
- Speech Enhancement (SE) Block: Utilizes CMGAN, a conformer-based metric GAN, to generate an enhanced version of the input audio ($W_{en}$).
- SNR-level Detection Block: This block calculates the cosine similarity between the original spectrogram and the enhanced spectrogram. High similarity indicates a clean signal (where the SE model intervenes minimally), while low similarity indicates a noisy signal. This similarity score ($S$) is transformed into a clamped SNR-level score ($S'$).
- Waveform Reconstitution: Instead of a binary choice, the system creates a hybrid signal ($W_{re}$) using the formula: $W_{re} = W_{in} \times S' + W_{en} \times (1 - S')$. This ensures that clean speech remains largely untouched while noisy speech is processed more heavily.
- Emotion Recognition Block: A multi-task model based on HuBERT predicts Arousal, Valence, Dominance, and ten categorical emotions.
Experimental Setup
The system was evaluated using the MSP-PODCAST (v1.10) dataset for naturalistic speech. Background noise from AudioSet was used to contaminate clean speech at various Signal-to-Noise Ratios (SNRs). The SE model was pre-trained on Voice Bank+DEMAND, and LibriSpeech served as an out-of-domain test set to verify the SNR detector's generalization to high-quality clean speech.
Results and Discussion
NRSER consistently outperformed or matched baselines across various noise levels. A key advantage of the system is its ability to mitigate the "SE tax"—the performance drop typically seen when applying enhancement to clean audio. On clean MSP data, NRSER maintained an F1 score nearly identical to the clean-only baseline, whereas standard static SE systems saw a noticeable drop.
As shown in the results, NRSER provides significant gains in Valence and F1 scores for categorical emotions compared to fixed preprocessing steps. The SNR-level detector also proved highly effective at filtering "pure noise," achieving over 90% accuracy in identifying non-speech segments at a threshold of 0.6. This prevents the system from making erroneous emotion predictions on background noise alone.
Conclusion
NRSER effectively improves the noise robustness of SER systems without sacrificing performance on clean speech. While highly effective for Valence and Category recognition, the authors noted a slight decrease in effectiveness for Arousal and Dominance, potentially due to SE removing subtle acoustic cues. Future work will explore larger architectures like hubert-large and broader applications of the SNR-level detection block for automated data selection.
Summary updated