Correlated-spaces regression for learning continuous emotion dimensions

Mihalis A. Nicolaou, Stefanos Zafeiriou, Maja Pantic

Introduction

In the field of affective computing, there is a significant shift from classifying discrete emotional categories (e.g., "happy" vs. "sad") toward modeling continuous emotion dimensions. These dimensions, such as valence (pleasantness) and arousal (activation), provide a more nuanced representation of human affect. However, automated recognition of these dimensions faces two primary hurdles: the weak correlation between raw sensory features (audio/video) and abstract emotional states, and the tendency of models to ignore the inherent inter-dependencies between different emotional dimensions.

This paper introduces Correlated-Spaces Regression (CSR), a framework designed to exploit the fact that emotional dimensions are often more strongly correlated with one another than they are with raw pixels or audio waves. By leveraging these inter-correlations, CSR "heals" the gap between features and labels, significantly improving predictive accuracy.

The Inter-dependency Hypothesis

Before proposing their model, the authors conducted an empirical study using Relevance Vector Machines (RVM) to determine which signals best predict a specific emotion dimension. They compared raw facial features, audio features, and the values of other emotion dimensions.

The findings were striking: to predict a dimension like "Valence," knowing the subject's "Arousal" or "Power" scores was more informative than analyzing their facial expressions or voice. This suggests that human emotions exist within a structured dimensional space where the position on one axis heavily informs the position on others.

Methodology: Correlated-Spaces Regression

To capitalize on these relationships, the authors developed the CSR framework based on Canonical Correlation Analysis (CCA). Unlike standard regression, which maps features directly to labels, CSR projects both into a shared latent space.

The Correlated-Spaces Regression (CSR) framework for training and testing.

The CSR process involves four critical stages:

  1. Projection: Multi-modal features (audio and video) and labels (emotion dimensions) are projected into a common latent space using learned projection vectors ($V_{F}$ and $V_{R}$).
  2. Correlation Maximization: During training, the model maximizes the correlation between the projected feature space and the label space.
  3. Label Uncorrelation: The labels are mathematically uncorrelated in the projected space, simplifying the subsequent regression task.
  4. Mapping: A function $f$ is learned to map the projected features to the new, uncorrelated label space. During testing, predictions are projected back into the original label space.

Experimental Evaluation

The study utilized the SEMAINE database, a benchmark consisting of audio-visual recordings of human-agent interactions. The data includes annotations for five dimensions: Valence, Arousal, Power, Expectation, and Intensity. Features included 226-dimensional facial vectors (AAM-based) and 58-dimensional audio vectors (MFCCs, pitch, energy).

Key Results

CSR consistently outperformed standard feature-level fusion and single-cue regression baselines. The model demonstrated a "healing effect," where the forced correlation in the latent space allowed the model to capture the underlying emotional structure more effectively than traditional methods.

Performance comparison of CSR against single-cue and fusion baselines across continuous dimensions.

As shown in the results table, the CSR method ($R_{s \setminus k}$) achieved the highest correlation coefficients (COR) across all dimensions:

  • Arousal: CSR boosted the COR to 0.47, nearly matching the performance of using ground-truth dimensions as predictors.
  • Valence: Multi-modal CSR achieved a COR of 0.21, a significant improvement over the 0.16 achieved by standard fusion techniques.
  • Dimensionality Reduction: The framework effectively compressed high-dimensional facial data into a potent representation without sacrificing predictive power.

Furthermore, the authors noted that Anger was the most "dimensional" of the basic emotions, showing the strongest alignment with the valence-arousal-power framework.

Conclusion and Future Work

Correlated-Spaces Regression provides a robust methodology for multi-modal fusion in affective computing. By treating emotional dimensions as an interconnected system rather than isolated variables, the model achieves superior performance on challenging continuous datasets.

Future research may focus on incorporating temporal dynamics (e.g., via LSTMs) to model the flow of emotion over time, as the current CSR model operates on discrete segments. Additionally, testing the "uncorrelated label space" theory on broader datasets like RECOLA or AVEC remains a priority for validating the generalizability of the approach.

Summary updated

Abstract

Adopting continuous dimensional annotations for affective analysis has been gaining rising attention by researchers over the past years. Due to the idiosyncratic nature of this problem, many subproblems have been identified, spanning from the fusion of multiple continuous annotations to exploiting output-correlations amongst emotion dimensions. In this paper, we firstly empirically answer several important questions which have found partial or no answer at all so far in related literature. In more detail, we study the correlation of each emotion dimension (i) with respect to other emotion dimensions, (ii) to basic emotions (e.g., happiness, anger). As a measure for comparison, we use video and audio features. Interestingly enough, we find that (i) each emotion dimension is more correlated with other emotion dimensions rather than with face and audio features, and similarly (ii) that each basic emotion is more correlated with emotion dimensions than with audio and video features. A similar conclusion holds for discrete emotions which are found to be highly correlated to emotion dimensions as compared to audio and/or video features. Motivated by these findings, we present a novel regression algorithm (Correlated-Spaces Regression, CSR), inspired by Canonical Correlation Analysis (CCA) which learns output-correlations and performs supervised dimensionality reduction and multimodal fusion by (i) projecting features extracted from all modalities and labels onto a common space where their inter-correlation is maximised and (ii) learning mappings from the projected feature space onto the projected, uncorrelated label space.
BibTeX
@inproceedings{nicolaou2013correlated,
  author = {Mihalis A. Nicolaou and Stefanos Zafeiriou and Maja Pantic},
  title = {{Correlated-spaces regression for learning continuous emotion dimensions}},
  booktitle = {ACM Multimedia},
  year = {2013},
  doi = {10.1145/2502081.2502201},
}
Figure