Speech Emotion Recognition Based on Multi-Label Emotion Existence Model

Atsushi Ando, Ryo Masumura, Hosana Kamiyama, Satoshi Kobashikawa, Yushi AonoInterspeech 2019

TL;DR

This paper introduces a novel approach to speech emotion recognition (SER) by modeling the "existence" of multiple emotions within an utterance, rather than just focusing on a single dominant emotion. This "Multi-Label Emotion Existence" (MLEE) model, trained in a two-step process, significantly improves dominant emotion recognition accuracy by better handling the inherent ambiguity of human emotional expression in speech.

What Problem Does This Paper Address?

Automatic Emotion Recognition (AER) aims to identify emotions from speech, with applications ranging from customer service to human-robot interaction. A key challenge in AER is that human speech often doesn't contain just one clear emotion. Instead, utterances can express multiple emotions simultaneously, or have subtle underlying emotional cues. Traditional AER methods often assume a single "dominant" emotion as the ground truth, which can be an oversimplification. While some methods attempt to address this by using "soft targets" (representing the proportion of annotations for each emotion) or by specifying secondary emotions, these approaches can be complex to train or may not fully leverage the nuanced emotional information present. This paper tackles the problem of accurately recognizing the dominant emotion by explicitly modeling the presence or absence of all potential emotions within an utterance, hypothesizing that this is an easier and more informative task than directly estimating emotion proportions.

Methods

The core innovation is the Multi-Label Emotion Existence (MLEE) model. Instead of predicting a single dominant emotion, MLEE predicts the probability of each emotion being present or absent in an utterance. This is framed as a multi-label classification problem.

The proposed method employs a two-step training strategy:

  1. MLEE Pre-training: An MLEE model is trained first. This model uses a standard CNN-BLSTM-Attention architecture (similar to conventional SER models) but with a sigmoid activation function in the final layer to output probabilities for each emotion's existence. This step focuses on learning features that capture the presence/absence of individual emotions.
  2. Dominant Emotion Recognition Fine-tuning: The pre-trained MLEE model's encoder is then used to fine-tune a dominant emotion recognition model. This can be done in a few ways:
  3. MLEE only: The MLEE model's output (emotion existence probabilities) is directly used to infer the dominant emotion by selecting the class with the highest existence probability.
  4. MLEE pre-training: The MLEE encoder is frozen, and its output is fed into a classifier to predict the dominant emotion. This leverages the learned emotion existence features.
  5. MLEE pre-training fusion: This approach combines the pre-trained MLEE encoder with a pre-trained encoder for dominant emotion recognition. The outputs of both encoders are concatenated, providing a richer feature representation for the final dominant emotion classifier.

This two-step approach allows the model to first understand the emotional landscape of an utterance (which emotions are present) before making a final decision on the dominant one.

Datasets

The experiments were conducted on the Interactive Emotional Dyadic Motion Capture (IEMOCAP) database. This database contains approximately 12 hours of dialog speech from 10 speakers. For their experiments, the authors focused on improvised speech sessions and considered four target emotions: neutral (neu), happy (hap), sad (sad), and angry (ang). They created two training sets:

  • Dominant: Utterances with a single, clearly annotated dominant emotion.
  • All: This set includes utterances with a dominant emotion plus those where one of the target emotions was present but not necessarily dominant. This "all" set is crucial for MLEE, as it can utilize more data by treating utterances without a clear dominant target emotion as negative samples for existence prediction.

Key Insights and Ideas

  • Emotion Ambiguity is Key: The paper strongly argues that human emotions in speech are rarely singular. Explicitly modeling this ambiguity is more effective than forcing a single-label prediction.
  • Existence vs. Proportion: Estimating the presence/absence of an emotion is hypothesized to be an easier and more robust task for a model than directly estimating the proportion or intensity of each emotion.
  • Auxiliary Task Power: Using emotion existence prediction as an auxiliary task significantly boosts the performance of the primary task (dominant emotion recognition).
  • Data Augmentation for MLEE: MLEE can leverage a larger portion of the annotated data (the "all" set) compared to traditional methods that discard utterances without a clear dominant emotion. This increased data availability leads to better model estimation.
  • Complementary Features: The "MLEE pre-training fusion" method suggests that features learned for emotion existence and features learned for dominant emotion prediction are complementary, and combining them yields the best results.

Key Results

The proposed MLEE-based methods consistently outperformed conventional dominant emotion recognition models (both hard-target and soft-target) across the board.

  • MLEE pre-training fusion (soft-target) achieved the highest weighted accuracy (WA) of 66.1% and unweighted accuracy (UA) of 65.4%.
  • Even the simpler MLEE only model performed comparably to soft-target training, demonstrating the inherent value of the MLEE approach.
  • The paper also shows that using the "all" training set (which includes more data) generally leads to better performance than using only the "dominant" set, especially for MLEE methods.

Limitations and Future Work

  • Limited Emotion Set: The experiments focused on only four target emotions (neutral, happy, sad, angry). The authors acknowledge that real-world emotional expression is far more complex and involves a wider range of emotions.
  • Dataset Specificity: The results are based on the IEMOCAP database. Further evaluation on other corpora is needed to confirm the generalizability of the proposed method.

Future work suggested includes:

  • Evaluating the MLEE approach on other emotion recognition corpora.
  • Exploring more complex network architectures or fusion strategies.
  • Investigating how to better model the intensity or proportion of emotions once their existence is established.

Summary updated

BibTeX
@inproceedings{ando2019speech,
  author = {Atsushi Ando and Ryo Masumura and Hosana Kamiyama and Satoshi Kobashikawa and Yushi Aono},
  title = {{Speech Emotion Recognition Based on Multi-Label Emotion Existence Model}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure