Two-level Attention with Two-stage Multi-task Learning for Facial Emotion Recognition

Xiaohua Wang, Muzi Peng, Lijuan Pan, Min Hu, Chunhua Jin, Fuji Ren

Abstract

Facial emotion recognition (FER) in real-world environments is complicated by variations in lighting, pose, and background. While categorical models are common, dimensional models (Valence and Arousal) offer a more nuanced representation of human affect. This paper presents 2Att-2Mt, a framework that combines a two-level attention mechanism with two-stage multi-task learning. By capturing both spatial and layer-wise features and utilizing categorical labels as auxiliary signals, the proposed method significantly improves dimensional emotion estimation, particularly for the challenging task of arousal prediction.

Introduction

Human emotions are complex and often better described by continuous dimensions—Valence (positivity/negativity) and Arousal (intensity)—than by discrete categories. However, estimating these values from static "in-the-wild" images is difficult. Subtle emotional cues are often obscured by non-emotional factors like race, illumination, and background noise. Furthermore, dimensional models are inherently harder to train than categorical ones. To address these issues, the 2Att-2Mt framework introduces structural innovations to focus on relevant facial regions and leverage the correlation between different emotion representations.

Methodology

Two-level Attention Mechanism

The core of the 2Att-2Mt framework is its dual-attention strategy designed to extract robust features from noisy images.

  1. First-level (Position-level) Attention: The model employs Residual Attention Blocks. Each block contains a "trunk branch" for standard feature extraction and a "mask branch" that generates soft weights. These weights highlight critical facial regions (e.g., eyes, mouth) while suppressing background noise.
The architecture of the Residual Attention Block used for position-level attention.
  1. Second-level (Layer-level) Attention: This novel component treats feature vectors from different CNN layers as a sequence. A Bi-directional Recurrent Neural Network (Bi-RNN) with self-attention processes this sequence, allowing the model to adaptively learn the relationships between global high-level features and local low-level details.

Two-stage Multi-task Learning

The framework utilizes a two-stage learning process to improve convergence and accuracy:

  • Stage 1: The model is trained on both categorical and dimensional tasks simultaneously. Categorical labels serve as an auxiliary task, providing diverse characteristics that help the model learn more effective representations for continuous estimation.
  • Stage 2: The model focuses on joint Valence and Arousal estimation, exploiting the high correlation between these two dimensions.
Overview of the 2Att-2Mt framework, showing the two-level attention and multi-task learning stages.

Robust Loss Function

To mitigate the impact of noisy or subjective labels in large-scale datasets, the authors replace the standard Mean Square Error (MSE) with Tukey’s biweight function. This robust loss function reduces the influence of outliers (samples with large errors), leading to more stable training on "in-the-wild" data.

Experiments and Results

The framework was evaluated on the AffectNet dataset, which contains approximately 300,000 images with diverse environmental conditions. Performance was measured using the Concordance Correlation Coefficient (CCC) and Root Mean Square Error (RMSE).

The 2Att-2Mt model demonstrated significant improvements over baseline methods:

Method Valence (CCC) Arousal (CCC) Mean CCC
SVR 0.372 0.182 0.277
CNN 0.600 0.340 0.470
2Att-2Mt (Ours) 0.714 0.556 0.635

Notably, the model achieved a substantial gain in Arousal prediction, which is traditionally much harder to capture than Valence. Ablation studies indicated that a two-block attention structure provides the optimal balance between depth and feature preservation.

Conclusion

The 2Att-2Mt framework effectively addresses the challenges of dimensional facial emotion recognition in unconstrained environments. By combining spatial and layer-wise attention with a multi-task learning strategy, the model achieves state-of-the-art results on the AffectNet dataset. Future work will focus on multi-domain strategies to leverage multiple datasets and exploring higher-resolution inputs to capture even finer emotional nuances.

Summary updated

BibTeX
@article{wang2019two,
  author = {Xiaohua Wang and Muzi Peng and Lijuan Pan and Min Hu and Chunhua Jin and Fuji Ren},
  title = {{Two-level Attention with Two-stage Multi-task Learning for Facial Emotion Recognition}},
  journal = {J. Vis. Commun. Image Represent.},
  year = {2019},
  url = {https://openreview.net/forum?id=Bku5jyMkNJ},
}
Figure