Introduction
In the field of affective computing, there is a significant shift from classifying discrete emotional categories (e.g., "happy" vs. "sad") toward modeling continuous emotion dimensions. These dimensions, such as valence (pleasantness) and arousal (activation), provide a more nuanced representation of human affect. However, automated recognition of these dimensions faces two primary hurdles: the weak correlation between raw sensory features (audio/video) and abstract emotional states, and the tendency of models to ignore the inherent inter-dependencies between different emotional dimensions.
This paper introduces Correlated-Spaces Regression (CSR), a framework designed to exploit the fact that emotional dimensions are often more strongly correlated with one another than they are with raw pixels or audio waves. By leveraging these inter-correlations, CSR "heals" the gap between features and labels, significantly improving predictive accuracy.
The Inter-dependency Hypothesis
Before proposing their model, the authors conducted an empirical study using Relevance Vector Machines (RVM) to determine which signals best predict a specific emotion dimension. They compared raw facial features, audio features, and the values of other emotion dimensions.
The findings were striking: to predict a dimension like "Valence," knowing the subject's "Arousal" or "Power" scores was more informative than analyzing their facial expressions or voice. This suggests that human emotions exist within a structured dimensional space where the position on one axis heavily informs the position on others.
Methodology: Correlated-Spaces Regression
To capitalize on these relationships, the authors developed the CSR framework based on Canonical Correlation Analysis (CCA). Unlike standard regression, which maps features directly to labels, CSR projects both into a shared latent space.
The CSR process involves four critical stages:
- Projection: Multi-modal features (audio and video) and labels (emotion dimensions) are projected into a common latent space using learned projection vectors ($V_{F}$ and $V_{R}$).
- Correlation Maximization: During training, the model maximizes the correlation between the projected feature space and the label space.
- Label Uncorrelation: The labels are mathematically uncorrelated in the projected space, simplifying the subsequent regression task.
- Mapping: A function $f$ is learned to map the projected features to the new, uncorrelated label space. During testing, predictions are projected back into the original label space.
Experimental Evaluation
The study utilized the SEMAINE database, a benchmark consisting of audio-visual recordings of human-agent interactions. The data includes annotations for five dimensions: Valence, Arousal, Power, Expectation, and Intensity. Features included 226-dimensional facial vectors (AAM-based) and 58-dimensional audio vectors (MFCCs, pitch, energy).
Key Results
CSR consistently outperformed standard feature-level fusion and single-cue regression baselines. The model demonstrated a "healing effect," where the forced correlation in the latent space allowed the model to capture the underlying emotional structure more effectively than traditional methods.
As shown in the results table, the CSR method ($R_{s \setminus k}$) achieved the highest correlation coefficients (COR) across all dimensions:
- Arousal: CSR boosted the COR to 0.47, nearly matching the performance of using ground-truth dimensions as predictors.
- Valence: Multi-modal CSR achieved a COR of 0.21, a significant improvement over the 0.16 achieved by standard fusion techniques.
- Dimensionality Reduction: The framework effectively compressed high-dimensional facial data into a potent representation without sacrificing predictive power.
Furthermore, the authors noted that Anger was the most "dimensional" of the basic emotions, showing the strongest alignment with the valence-arousal-power framework.
Conclusion and Future Work
Correlated-Spaces Regression provides a robust methodology for multi-modal fusion in affective computing. By treating emotional dimensions as an interconnected system rather than isolated variables, the model achieves superior performance on challenging continuous datasets.
Future research may focus on incorporating temporal dynamics (e.g., via LSTMs) to model the flow of emotion over time, as the current CSR model operates on discrete segments. Additionally, testing the "uncorrelated label space" theory on broader datasets like RECOLA or AVEC remains a priority for validating the generalizability of the approach.
Summary updated