Spectro-temporal modulations for robust speech emotion recognition

Lan-Ying Yeh, Tai-Shih ChiInterspeech 2010

TL;DR

This paper introduces a novel set of "spectro-temporal modulation" features derived from an auditory model to improve speech emotion recognition, especially in noisy conditions. These features outperform traditional MFCCs and prosodic features when tested on mismatched clean training and noisy testing data, demonstrating greater robustness to real-world acoustic challenges.

What Problem Does This Paper Address?

Recognizing emotions in speech is crucial for more natural human-computer interaction. However, most existing research focuses on clean speech, which is unrealistic. Real-world applications often involve noisy environments, and current methods struggle to maintain accuracy when speech is corrupted by background noise. While some studies attempt to address this by using a vast number of features, they often fail when the noise characteristics change between training and testing (mismatched conditions). This paper aims to develop features that are inherently robust to varying noise conditions, even when the noise type or level is unknown during training.

Methods

The core of this work is the extraction of "spectro-temporal modulation" (RS) features. These features are inspired by how the human auditory system processes sound. The proposed method uses a two-stage auditory model:

  1. Cochlear Module: This module mimics the frequency filtering and non-linear processing of the inner ear, producing an "auditory spectrogram."
  2. Cortical Module: This module analyzes the auditory spectrogram to extract features related to spectro-temporal modulations. Specifically, it models how neurons in the auditory cortex respond to changes in sound over time and frequency. These responses are characterized by "rate" (how fast modulations occur) and "scale" (how broad they are across frequencies).

The extracted RS features are then used with a linear-kernel Support Vector Machine (SVM) for emotion classification. To further improve efficiency and robustness, a Sequential Forward Floating Selection (SFFS) method is employed for dimensionality reduction, identifying the most informative RS features. These RS features are compared against a standard set of Mel-Frequency Cepstral Coefficients (MFCCs) combined with prosodic features.

Datasets

  • Berlin Emotional Speech Database (EMO-DB): This is the primary dataset used. It contains acted emotions (anger, happiness, sadness, fear, disgust, boredom, neutral) from 10 German speakers, recorded under studio conditions. Utterances with subjective recognition rates below 80% were excluded.
  • NOISEX-92 Database: This database was used to generate noisy speech by adding white and babble noises to the clean EMO-DB samples at various Signal-to-Noise Ratios (SNRs).

Key Insights and Ideas

  • Auditory Inspiration for Robustness: Features derived from an auditory model, specifically focusing on spectro-temporal modulations, show inherent robustness to noise. This suggests that mimicking human auditory processing can lead to more resilient speech recognition systems.
  • Redundancy in RS Features: The SFFS analysis revealed significant redundancy within the initial set of RS features. This indicates that a smaller, carefully selected subset of RS features can achieve comparable or even better performance, making the system more efficient.
  • Mismatched Condition Performance: The key finding is that RS features significantly outperform traditional MFCCs + prosodic features in mismatched conditions (clean training, noisy testing), particularly at low SNRs. This highlights their practical value for real-world applications.
  • Noise Type Matters: While RS features are generally robust, the paper notes that babble noise, which has modulations similar to speech, requires higher resolution in the RS domain compared to white noise.

Key Results

  • Superior Performance in Noise: In mismatched conditions (clean train, noisy test), the proposed RS features consistently outperformed MFCCs + prosodic features across various SNRs (20dB down to 0dB) for both white and babble noise.
  • Low SNR Advantage: The performance gap between RS features and MFCCs + prosodic features widened significantly at low SNRs (<10dB), where RS features maintained much higher recognition rates.
  • Dimensionality Reduction: After dimensionality reduction using SFFS, the RS92 features (92 features) still demonstrated strong performance, comparable to the original RS180 features, especially in white noise.
  • Example Performance: In 0dB babble noise, RS180 achieved a recognition rate of 49.46%, while MFCC156+PRO30 dropped to 37.53%. Similarly, RS92 achieved 50.01% in 0dB babble noise, outperforming MFCC78+PRO30 (42.58%) and MFCC52+PRO30 (42.58%).

Limitations and Future Work

  • Acted Emotions: The EMO-DB database contains acted emotions, which may not fully represent natural emotional expressions.
  • Future Work: The authors plan to extend their research to larger databases containing natural emotions, such as the FAU Aibo Emotion Corpus, to further validate the robustness of their spectro-temporal modulation features.

Summary updated

BibTeX
@inproceedings{yeh2010spectro,
  author = {Lan-Ying Yeh and Tai-Shih Chi},
  title = {{Spectro-temporal modulations for robust speech emotion recognition}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure