Abstract
Analyzing human affect is vital for human-computer interaction systems. Most methods are developed in restricted scenarios which are not practical for in-the-wild settings. The Affective Behavior Analysis in-the-wild (ABAW) 2021 Contest provides a benchmark for this in-the-wild problem. In this paper, we introduce a multi-modal and multi-task learning method by using both visual and audio information. We use both AU and expression annotations to train the model and apply a sequence model to further extract associations between video frames. We achieve an AU score of 0.712 and an expression score of 0.477 on the validation set. These results demonstrate the effectiveness of our approach in improving model performance.
Introduction
While AI has become proficient at recognizing faces in controlled laboratory settings, understanding human affect in the real world—"in-the-wild"—remains a significant challenge. Real-world videos, such as those found on YouTube, suffer from varied lighting, extreme head poses, occlusions, and severe data imbalance where certain emotions are rarely captured.
Furthermore, multi-task learning in this field is often hindered by incomplete annotations. In large datasets like Aff-Wild2, a specific video frame might be labeled for Action Units (AUs) but lack a label for the overall facial expression, or vice versa. This research proposes a framework that learns from both visual and audio modalities and both tasks (AU and expression) simultaneously, despite these data gaps.
Methodology
The proposed pipeline integrates visual, audio, and temporal information through a three-stage process:
- Multi-Task Visual Backbone: The model utilizes an IResNet100 backbone pre-trained on the Glint360K face recognition dataset. To handle missing labels, a Rotation Training strategy is employed, alternating updates between the AU and Expression heads either epoch-by-epoch or batch-by-batch depending on label availability.
- Audio Stream: Recognizing that sound carries emotional weight, Mel spectrograms are extracted from the audio and processed using a Time Delay Neural Network (TDNN).
- Sequence Modeling: To capture temporal evolution, visual and audio features are aligned and fed into a Transformer encoder. This allows the model to analyze a window of 30 frames to make predictions, providing context that isolated frame analysis lacks.
To address class imbalance, the authors employed Weighted Binary Cross Entropy for AUs and Focal Loss for expressions, prioritizing "hard" or rare examples.
Datasets and Training
- Aff-Wild2: The primary benchmark, containing 548 videos and over 2.8 million frames collected from YouTube.
- BP4D: Supplementary training data used to bolster performance on rare Action Units (e.g., AU15, AU23, AU24).
- Glint360K: Used for initial pre-training of the visual backbone to ensure high-quality facial feature extraction.
Key Results
The proposed method demonstrated significant improvements over the ABAW2 competition baselines on the validation set. The AU score saw a dramatic jump, more than doubling the baseline performance, likely due to the multi-task learning approach and the inclusion of the BP4D dataset.
| Task | Metric | Baseline | Ours |
|---|---|---|---|
| Action Unit (AU) | Avg F1 + Accuracy | 0.310 | 0.712 |
| Expression | Avg F1 + Accuracy | 0.366 | 0.477 |
Conclusion
By leveraging face recognition as a foundation, incorporating underutilized audio signals, and utilizing temporal context via Transformers, this multi-modal framework effectively addresses the complexities of in-the-wild affect recognition. The rotation training strategy provides a practical solution for handling incomplete labels in real-world datasets. Future work will include ablation studies to further quantify the individual contributions of the audio stream and the Transformer-based sequence modeling.
Summary updated