A Speech Emotion Recognition Framework for Better Discrimination of Confusions

Jiawang Liu, Haoxiang Wang2021 · DOI

TL;DR

This paper introduces a novel framework for speech emotion recognition (SER) that tackles the common problem of misclassifying similar emotions. By combining a CNN-LSTM architecture with a triplet loss function and data augmentation, the model learns to better distinguish between confusable emotional states, achieving state-of-the-art results on the IEMOCAP dataset.

What Problem Does This Paper Address?

Speech emotion recognition (SER) is crucial for natural human-machine interaction, but a persistent challenge is the accurate differentiation of emotions that sound similar. For instance, "happy" and "neutral" emotions are frequently misclassified. This confusion arises from subtle acoustic differences and imbalanced datasets, where majority classes can bias model predictions. Existing SER methods often struggle with these confusable pairs, limiting their real-world applicability.

Methods

The proposed framework builds upon a CNN-LSTM architecture, a common choice for SER due to its ability to capture both spatial (frequency-based) and temporal (time-series) features in speech.

Here's what's novel:

  • Triplet Loss: Instead of just classifying emotions, the model is trained to learn embeddings where utterances of the same emotion are pulled closer together, while utterances of different emotions are pushed further apart. Specifically, it uses a triplet loss function that aims to make an "anchor" utterance more similar to a "positive" (same emotion) utterance than to a "negative" (different emotion) utterance. This encourages the model to focus on the subtle distinctions between confusable emotions.
  • Triplet Selection: To optimize training, the paper employs a strategy to select "hard" and "semi-hard" triplets. These are triplets where the negative sample is only slightly further away than the positive sample, forcing the model to learn finer-grained distinctions. This also helps reduce computational complexity.
  • Data Augmentation (SpecAugment): To address the issue of imbalanced datasets, the researchers leverage SpecAugment. This technique, originally developed for Automatic Speech Recognition (ASR), applies time warping, frequency masking, and time masking to the Mel-Frequency Cepstral Coefficients (MFCCs). This artificially enriches the minority emotion classes, helping to balance the dataset and improve the effectiveness of the triplet loss.
  • Joint Loss: The final loss function is a combination of the triplet loss and the standard cross-entropy loss, weighted by parameters $\alpha$ and $(1-\alpha)$. This allows the model to benefit from both discriminative embedding learning and direct classification.

Datasets

The primary dataset used for training and evaluation is the IEMOCAP (Interactive Emotional Dyadic Motion Capture) dataset.

  • IEMOCAP: This dataset contains approximately 12 hours of speech data from scripted and improvised scenarios, featuring two speakers. It is annotated with 10 emotion classes. For consistency with prior work and to focus on particularly confusable emotions, the authors consider four specific emotions: happy, angry, sad, and neutral, derived from improvised utterances, with "excitement" being replaced by "happy."

Key Insights and Ideas

  • The Power of "Hard" Examples: Focusing training on triplets where distinguishing between emotions is difficult (hard/semi-hard triplets) significantly improves the model's ability to discriminate between confusable classes.
  • Data Augmentation as a Balancer: SpecAugment proves effective in mitigating the negative impact of imbalanced datasets on SER performance, especially when combined with triplet loss. It helps the model generalize better by creating more diverse training examples for underrepresented emotions.
  • Joint Loss Synergy: Combining triplet loss with cross-entropy loss allows the model to learn both discriminative feature representations and accurate classification, leading to superior performance compared to using either loss alone.
  • Confusable Pairs are the Bottleneck: The paper explicitly identifies and targets the misclassification of specific emotion pairs (e.g., happy vs. neutral) as the key area for improvement.

Key Results

  • The proposed framework achieved 79.52% Weighted Accuracy (WA) and 78.30% Unweighted Accuracy (UA) on the IEMOCAP dataset.
  • This represents a significant improvement over previous state-of-the-art models, with an increase of over 3.34% on WA and 1.94% on UA compared to other recent methods.
  • Ablation studies showed that the triplet loss alone improved WA by 0.47% and UA by 0.94% compared to a CNN-LSTM without triplet loss.
  • The data augmentation further boosted performance, increasing WA by 1.67% and UA by 1.52% over the model without augmentation.
  • Confusion matrices clearly illustrate that the proposed method reduces misclassifications, particularly between angry, sad, and happy emotions.

Limitations and Future Work

  • Artificial Data Enhancement: The authors suggest exploring more advanced artificial data enhancement techniques, such as Generative Adversarial Networks (GANs), to generate even more diverse and realistic emotional utterances for training.
  • Generalization to Other Datasets: While the results on IEMOCAP are strong, future work could investigate the framework's performance on other SER datasets to assess its generalizability.
  • Exploring Other Confusable Pairs: The current work focuses on specific confusable pairs. Further research could systematically analyze and address other challenging emotion distinctions.

Summary updated

BibTeX
@misc{liu2021speech,
  author = {Jiawang Liu and Haoxiang Wang},
  title = {{A Speech Emotion Recognition Framework for Better Discrimination of Confusions}},
  year = {2021},
  doi = {10.21437/interspeech.2021-718},
}
Figure