Abandoning emotion classes - towards continuous emotion recognition with modelling of long-range dependencies

Martin Wöllmer, Florian Eyben, Stephan Reiter, Björn Schuller, Cate Cox, Ellen Douglas-Cowie, Roddy CowieInterspeech 2008

TL;DR

This paper proposes a novel approach to continuous emotion recognition in speech by modeling long-range dependencies, moving beyond discrete emotion classes. By using Long Short-Term Memory Recurrent Neural Networks (LSTM-RNNs) and Conditional Random Fields (CRFs), the authors achieve state-of-the-art performance, particularly in recognizing "activation," matching human annotator accuracy.

What Problem Does This Paper Address?

Traditional emotion recognition systems often rely on classifying speech into discrete emotion categories (e.g., happy, sad, angry). However, human emotions are far more nuanced and exist on a continuous spectrum, often described by dimensions like "valence" (pleasantness) and "activation" (arousal). This discrete approach limits the practical applicability of these systems. The authors aim to bridge this gap by developing a method for continuous emotion recognition that can capture the subtle, evolving nature of emotions over time, acknowledging that emotional history plays a crucial role.

Methods

The paper introduces two primary methods for continuous emotion recognition:

  • Long Short-Term Memory Recurrent Neural Networks (LSTM-RNNs): This is the core novel approach. Unlike standard recurrent neural networks that struggle with remembering information over long periods, LSTM-RNNs are specifically designed to overcome this "vanishing gradient" problem. They achieve this through the use of "memory blocks" that can retain information for extended durations, allowing them to model the "emotional history" and long-range dependencies in speech. For this work, LSTM-RNNs are used to predict continuous values for valence and activation.
  • Conditional Random Fields (CRFs): As an alternative for discrete classification, CRFs are employed. CRFs are chosen because, like LSTM-RNNs, they do not assume independence between observations, making them suitable for modeling temporal dependencies. The continuous valence and arousal labels are quantized into discrete levels (4 and 7, respectively) for CRF classification.

Both methods utilize a rich set of acoustic features derived from Low-Level Descriptors (LLDs), including pitch, energy, spectral features, and voice quality. Hierarchical functionals are applied to these LLDs to create a fixed-size feature vector for each utterance, compensating for variable utterance lengths and statistical outliers.

Datasets

The primary dataset used is the Belfast Sensitive Artificial Listener (SAL) database, part of the HUMAINE project. This dataset is notable because it contains continuously labeled emotional dimensions (valence and activation) sampled at a high temporal resolution (10 ms). The authors used a subset of 25 recordings from 4 speakers, with continuous labels provided by 4 human annotators. The data was split into training (1,102 turns) and testing (590 turns) sets.

Key Insights and Ideas

  • Emotion is Continuous, Not Discrete: The paper strongly advocates for moving beyond discrete emotion classes, arguing that continuous dimensions like valence and activation better represent the complexity of human emotion.
  • Emotional History Matters: The core idea is that understanding an emotion requires considering not just the current acoustic signal but also the preceding emotional states. LSTM-RNNs and CRFs are shown to be effective in capturing these long-range dependencies.
  • Activation is Easier to Detect than Valence: Consistent with prior research, the results show that acoustic features alone are more effective for recognizing "activation" than "valence." This is attributed, in part, to the lack of linguistic information, which is crucial for discerning valence.
  • LSTM-RNNs Excel at Continuous Prediction: For continuous emotion recognition, LSTM-RNNs demonstrate superior performance over Support-Vector Regression (SVR), especially for activation.
  • CRFs Outperform SVMs for Discrete Classification: When applied to quantized emotion labels, CRFs, by modeling temporal dependencies, achieve better results than standard Support-Vector Machines (SVMs).

Key Results

  • Activation Recognition Matches Human Performance: For continuous "activation" recognition using LSTM-RNNs with the M+Nspk feature normalization, the Mean Squared Error (MSE) achieved was 0.08, which is reported as being as good as human annotation.
  • CRFs Improve Over SVMs: In discrete classification tasks, CRFs consistently outperformed SVMs, particularly when considering the best results for each classifier.
  • Valence Remains Challenging: Despite the advanced methods, valence recognition performance (both discrete and continuous) remained significantly lower than for activation, confirming its inherent difficulty from acoustic features alone.
  • Feature Normalization Impacts Performance: Different feature normalization strategies (e.g., MVSall, NRMall, M+Nall) showed varying degrees of success, with NRMall and M+Nall often yielding the best results for both classifiers.

Limitations and Future Work

  • Valence Detection Difficulty: The authors acknowledge that detecting valence from acoustic features alone is a hard task, even for humans.
  • Limited Dataset Size and Annotator Agreement: The SAL database, while valuable for its continuous labeling, has limitations: only 4 speakers and relatively low inter-annotator agreement. This can impact the robustness of the models.
  • Lack of Linguistic Information: The current approach relies solely on acoustic features. The authors suggest that integrating linguistic features (requiring speech recognition) is a promising avenue for improving valence detection.
  • Generalization to Larger Datasets: Future work should involve applying the developed concept of modeling long-range dependencies to larger, more diverse databases with quasi-continuous 3D emotion spaces.

Summary updated

BibTeX
@inproceedings{wollmer2008abandoning,
  author = {Martin Wöllmer and Florian Eyben and Stephan Reiter and Björn Schuller and Cate Cox and Ellen Douglas-Cowie and Roddy Cowie},
  title = {{Abandoning emotion classes - towards continuous emotion recognition with modelling of long-range dependencies}},
  booktitle = {Interspeech},
  year = {2008},
}
Figure