Abstract
Facial emotion recognition (FER) in real-world environments is complicated by variations in lighting, pose, and background. While categorical models are common, dimensional models (Valence and Arousal) offer a more nuanced representation of human affect. This paper presents 2Att-2Mt, a framework that combines a two-level attention mechanism with two-stage multi-task learning. By capturing both spatial and layer-wise features and utilizing categorical labels as auxiliary signals, the proposed method significantly improves dimensional emotion estimation, particularly for the challenging task of arousal prediction.
Introduction
Human emotions are complex and often better described by continuous dimensions—Valence (positivity/negativity) and Arousal (intensity)—than by discrete categories. However, estimating these values from static "in-the-wild" images is difficult. Subtle emotional cues are often obscured by non-emotional factors like race, illumination, and background noise. Furthermore, dimensional models are inherently harder to train than categorical ones. To address these issues, the 2Att-2Mt framework introduces structural innovations to focus on relevant facial regions and leverage the correlation between different emotion representations.
Methodology
Two-level Attention Mechanism
The core of the 2Att-2Mt framework is its dual-attention strategy designed to extract robust features from noisy images.
- First-level (Position-level) Attention: The model employs Residual Attention Blocks. Each block contains a "trunk branch" for standard feature extraction and a "mask branch" that generates soft weights. These weights highlight critical facial regions (e.g., eyes, mouth) while suppressing background noise.
- Second-level (Layer-level) Attention: This novel component treats feature vectors from different CNN layers as a sequence. A Bi-directional Recurrent Neural Network (Bi-RNN) with self-attention processes this sequence, allowing the model to adaptively learn the relationships between global high-level features and local low-level details.
Two-stage Multi-task Learning
The framework utilizes a two-stage learning process to improve convergence and accuracy:
- Stage 1: The model is trained on both categorical and dimensional tasks simultaneously. Categorical labels serve as an auxiliary task, providing diverse characteristics that help the model learn more effective representations for continuous estimation.
- Stage 2: The model focuses on joint Valence and Arousal estimation, exploiting the high correlation between these two dimensions.
Robust Loss Function
To mitigate the impact of noisy or subjective labels in large-scale datasets, the authors replace the standard Mean Square Error (MSE) with Tukey’s biweight function. This robust loss function reduces the influence of outliers (samples with large errors), leading to more stable training on "in-the-wild" data.
Experiments and Results
The framework was evaluated on the AffectNet dataset, which contains approximately 300,000 images with diverse environmental conditions. Performance was measured using the Concordance Correlation Coefficient (CCC) and Root Mean Square Error (RMSE).
The 2Att-2Mt model demonstrated significant improvements over baseline methods:
| Method | Valence (CCC) | Arousal (CCC) | Mean CCC |
|---|---|---|---|
| SVR | 0.372 | 0.182 | 0.277 |
| CNN | 0.600 | 0.340 | 0.470 |
| 2Att-2Mt (Ours) | 0.714 | 0.556 | 0.635 |
Notably, the model achieved a substantial gain in Arousal prediction, which is traditionally much harder to capture than Valence. Ablation studies indicated that a two-block attention structure provides the optimal balance between depth and feature preservation.
Conclusion
The 2Att-2Mt framework effectively addresses the challenges of dimensional facial emotion recognition in unconstrained environments. By combining spatial and layer-wise attention with a multi-task learning strategy, the model achieves state-of-the-art results on the AffectNet dataset. Future work will focus on multi-domain strategies to leverage multiple datasets and exploring higher-resolution inputs to capture even finer emotional nuances.
Summary updated