Towards Temporal Modelling of Categorical Speech Emotion Recognition

Wenjing Han, Huabin Ruan, Xiaohong Chen, Zhixiang Wang, Haifeng Li, Björn W. Schuller2018 · DOI

TL;DR

This paper tackles the challenge of temporal modeling in speech emotion recognition (SER) by proposing a novel sequence-to-sequence approach using LSTM-CTC. By hypothesizing that utterances contain both emotional and non-emotional segments, they automatically align emotional labels to relevant parts of speech, achieving state-of-the-art results on the IEMOCAP dataset.

What Problem Does This Paper Address?

Categorical Speech Emotion Recognition (SER) aims to classify the emotion conveyed in an utterance (e.g., happy, sad, angry). While speech is inherently sequential, a major hurdle in SER is the mismatch between frame-level acoustic features and utterance-level emotion labels. Traditionally, models aggregate frame-level information into a single utterance representation, losing valuable temporal dynamics. This paper addresses this by proposing a method to model SER temporally, aligning emotion labels to specific segments within an utterance rather than assigning a single label to the entire utterance.

Methods

The core of this paper's contribution lies in its novel sequence-to-sequence approach for SER, leveraging a LSTM-CTC (Connectionist Temporal Classification) model.

  1. Hypothesis and Label Sequence Construction: The authors hypothesize that utterances consist of emotional and non-emotional segments. Non-emotional segments are assumed to correspond to silent regions, short pauses, transitions between phonemes, and unvoiced phonemes. Based on this, they propose to transform the single utterance label into a sequence of labels, including a "Null" label for non-emotional segments. This sequence is constructed by first using text annotations and phonetic dictionaries to estimate the number of voiced phonemes (and thus potential emotional segments) in an utterance.
  2. LSTM-CTC for Temporal Alignment: The crucial step is to automatically align these hypothesized label sequences to the actual frames of the utterance. This is achieved using a LSTM-CTC model. CTC is a powerful technique that allows training sequence models without explicit frame-level alignment of labels, making it ideal for this problem. The LSTM processes the frame-level acoustic features, and the CTC loss function enables the model to learn to map the input sequence to the output label sequence (including "Null" labels) by implicitly learning the temporal boundaries of emotional segments.
  3. Sequence-to-Sequence Recipe: Unlike previous methods that often use a sequence-to-label approach (e.g., pooling LSTM outputs), this work adopts a true sequence-to-sequence recipe, aiming to decode the emotion label directly from the LSTM's output sequence.

Datasets

  • IEMOCAP (Interactive Emotional Dyadic Motion Capture) corpus: This is the primary dataset used for both training and evaluation. The authors mention using speaker-independent 10-fold cross-validation. They also note merging the "Excited" and "Happy" categories to align with previous work.

Key Insights and Ideas

  • Emotional vs. Non-Emotional Segments: The hypothesis that utterances contain distinct emotional and non-emotional segments is key. This allows for a more nuanced temporal modeling of emotion, acknowledging that not every moment in speech carries emotional valence.
  • CTC for Automatic Alignment: The application of CTC is a significant methodological choice. It elegantly solves the problem of needing precise frame-level annotations, which are difficult and time-consuming to obtain for emotions.
  • Phoneme-Level Segmentation Hypothesis: The finding that segmenting utterances based on the number of voiced phonemes (PN) yields the best results suggests that phonetic structure provides a meaningful basis for defining emotional segments. This is more effective than simply using word counts (WN) or double the phoneme count.
  • Beyond Energy: The paper highlights that their model doesn't solely rely on acoustic energy. Even frames with high energy don't automatically translate to high emotional output, indicating the model learns to capture more subtle emotional cues.

Key Results

  • State-of-the-Art Performance: The proposed LSTM-CTC method achieved the best results on the IEMOCAP corpus, with an Unweighted Average Recall (UAR) of 65.7% and a Weighted Average Recall (WAR) of 64.2%.
  • Superiority of Phoneme-Based Segmentation: The LSTM-CTC model performed best when the hypothesized number of emotional segments (yseg) was set to the number of voiced phonemes (PN) in the utterance. This outperformed setting yseg to the number of words (WN) or twice the number of voiced phonemes (PN*2).
  • Outperforming Baselines: The proposed method significantly outperformed baseline methods, including a feedforward DNN, frame-wise LSTM, final-pooling LSTM, and mean-pooling LSTM. It also showed superior performance compared to a previous CTC-style system from Lee et al. [12].

Limitations and Future Work

  • Manual Step: A limitation acknowledged is that the number of emotional segments (yseg) still needs to be specified, albeit guided by the hypothesis. While this is a significant reduction in manual effort compared to full frame-level annotation, it's not entirely automatic.
  • Future Work: The authors suggest exploring better strategies for extending utterance labels to label sequences and developing more effective sequence-to-sequence algorithms for categorical SER. This implies further research into refining the hypothesis or developing fully end-to-end methods for label sequence generation.

Summary updated

Abstract

To model the categorical speech emotion recognition task in a temporal manner, the first challenge arising is how to transfer the categorical label for each utterance into a label sequence.To settle this, we make a hypothesis that an utterance is consisting of emotional and non-emotional segments, and these non-emotional segments correspond to silent regions, short pauses, transitions between phonemes, unvoiced phonemes, etc.With this hypothesis, we propose to treat an utterance's label sequence as a chain of two states: the emotional state denoting the emotional frame and Null denoting the non-emotional frame.Then, we exploit a recurrent neural network based connectionist temporal classification model to automatically label and align an utterance's emotional segments with emotional labels, while non-emotional segments with Nulls.Experimental results on the IEMOCAP corpus validate our hypothesis and also demonstrate the effectiveness of our proposed method compared to the state-of-the-art algorithms.
BibTeX
@misc{han2018towards,
  author = {Wenjing Han and Huabin Ruan and Xiaohong Chen and Zhixiang Wang and Haifeng Li and Björn W. Schuller},
  title = {{Towards Temporal Modelling of Categorical Speech Emotion Recognition}},
  year = {2018},
  doi = {10.21437/interspeech.2018-1858},
}
Figure