TL;DR
This paper introduces EMMA and masked CoTEX, novel frameworks for multi-task affective behavior analysis. They leverage self-supervised learning with Masked Auto-Encoder (MAE) on unlabeled face images to extract rich facial prior knowledge, outperforming standard ImageNet pretraining and achieving top rankings in the ABAW4 challenge.
What Problem Does This Paper Address?
Accurately analyzing affective behavior, such as facial expressions, action units (AUs), and valence-arousal (VA) levels, is crucial for many applications. However, obtaining large, accurately annotated datasets for these tasks is incredibly labor-intensive and expensive. Existing deep learning models, while powerful, often struggle with generalization due to limited dataset sizes. This paper aims to overcome this by utilizing readily available unlabeled face images to learn robust facial prior knowledge through self-supervised learning.
Methods
The paper proposes two main methods:
- EMMA (Emotion Multi-Model Aggregation): This framework tackles multi-task affective behavior analysis (Expression Recognition - EXPR, AU Recognition - AU, and VA Regression - VA). It combines two key components:
- A Vision Transformer (ViT) pretrained using MAE on unlabeled face images. This ViT is used for AU and EXPR recognition.
-
A Convolutional Neural Network (CNN) pretrained on the AffectNet dataset (for expression recognition). This CNN is used to extract "pure and intact" expression features for VA regression. EMMA concatenates features from both the ViT (AU and EXPR scores) and the CNN (expression scores) to create a rich feature representation for VA regression. To prevent overfitting in the VA regression task, only the linear layer is finetuned.
-
Masked CoTEX (Co-Training for Expression Recognition): This method specifically targets EXPR. It employs a co-training framework using two parallel MAE-pretrained ViTs. To ensure these two "views" are complementary, a significant portion of image patches are randomly masked during training. The predictions from these two views are then encouraged to be consistent using Jensen-Shannon (JS) divergence. This approach aims to improve robustness and generalization for expression recognition.
Datasets
- Training:
- Unlabeled face images (for MAE pretraining).
- AffectNet (for CNN pretraining).
- CelebA (for ViT pretraining).
- EmotioNet (explored for pretraining, though with performance drops).
- Aff-Wild2 (used for training the MTL challenge).
- Synthetic face images generated from Aff-Wild2 (for LSD challenge training).
- Evaluation:
- ABAW4 challenge dataset, which includes:
- MTL (Multi-Task Learning) challenge: Utilizes static versions of Aff-Wild2, covering VA regression, EXPR (8 categories), and AU recognition (12 AUs).
- LSD (Learning from Synthetic Data) challenge: Focuses on EXPR using synthetic face images.
Key Insights and Ideas
- Facial Prior from MAE is Powerful: Pretraining ViTs with MAE on unlabeled face images yields significantly better performance on affective behavior tasks compared to standard ImageNet pretraining. This highlights the importance of learning intrinsic facial structure and relationships.
- Expression Scores as Pure VA Features: The paper finds that expression scores are particularly effective and "pure" features for VA regression, helping to avoid overfitting.
- Co-Training with Masking for Robustness: The masked CoTEX approach demonstrates that by creating independent views through random masking and enforcing consistency via JS divergence, robust expression recognition can be achieved.
- Mask Ratio and Batch Size Matter: For masked CoTEX, increasing the mask ratio and batch size significantly improves performance, suggesting that these parameters are crucial for effective co-training with masked inputs.
Key Results
- ABAW4 Challenge Performance: The proposed methods achieved strong results in the ABAW4 challenge, with the team ranking 2nd place in the MTL challenge and 4th place in the LSD challenge.
- EMMA's Superiority: EMMA, particularly when using MAE-pretrained ViT on CelebA, outperformed other methods in the MTL challenge, especially for the VA regression task.
- Masked CoTEX Effectiveness: Masked CoTEX demonstrated superior performance on the LSD challenge, with a mask ratio of 75% and a large batch size (1024) yielding the best results on the validation set.
Limitations and Future Work
- Overfitting with Excessive Masking: The paper notes that an excessively large mask ratio in masked CoTEX can lead to overfitting, as observed with a 95% mask ratio.
- Dataset Quality: The performance drop when using EmotioNet for pretraining suggests that the quality of face images (resolution, noise) in pretraining datasets can impact downstream task performance.
- Future Work: The authors suggest exploring MAE pretraining on different benchmarks and potentially investigating the impact of temporal information for more advanced affective behavior analysis. They also mention the potential of ensemble techniques for further performance gains.
Summary updated