Abstract
The COVID-19 pandemic necessitated the widespread use of face masks, presenting a significant challenge to automated Facial Expression Recognition (FER) systems. This paper evaluates the performance of FaceChannel, a lightweight convolutional neural network, in recognizing emotional affect (arousal and valence) from masked faces. We demonstrate that while standard models suffer significant performance degradation when the lower face is occluded, targeted fine-tuning allows the network to adapt by prioritizing the ocular region and interpreting mask deformations. Our findings suggest that FER systems can remain robust in constrained perception scenarios through efficient adaptation strategies.
Introduction
Facial Expression Recognition (FER) has traditionally relied on unconstrained, full-face imagery. State-of-the-art models often develop a "mouth-heavy" bias, utilizing features like smiles or open mouths to classify emotions. The mandatory use of face masks during the COVID-19 pandemic introduced a "constrained perception" problem, effectively blinding these systems to critical lower-face cues. This study quantifies this performance drop and explores how deep learning models can be adapted to recognize affect using primarily the upper face.
Methods
The researchers employed FaceChannel, a lightweight convolutional neural network (CNN) optimized for efficiency.
FaceChannel contains approximately 2 million parameters and utilizes "shunting inhibitory layers" to help the model adapt to various facial expressions. To evaluate mask impact, four training schemes were tested: baseline (trained on full faces), training from scratch on masked faces, partial fine-tuning of the final layer, and full fine-tuning of the entire network.
Datasets and Mask Generation
The study utilizes AffectNet, a large-scale dataset annotated for arousal and valence. To simulate pandemic conditions, the authors created MaskedAffectNet by programmatically applying synthetic surgical masks to AffectNet images using facial keypoint detection.
This methodology allowed for a direct comparison between masked and unmasked versions of identical emotional expressions, ensuring that performance differences were solely attributable to the occlusion.
Results
Performance was measured using the Concordance Correlation Coefficient (CCC). Results indicate a severe drop in baseline performance when models trained on full faces encounter masks, particularly for valence (positivity/negativity).
| Training Setup | Test Set | Arousal (CCC) | Valence (CCC) |
|---|---|---|---|
| Trained on Full Face | Full Face | 0.46 | 0.61 |
| Trained on Full Face | Masked Face | 0.18 | 0.16 |
| Fine-tuned (All Layers) | Masked Face | 0.45 | 0.53 |
Fine-tuning the model recovered nearly all performance for arousal and a substantial portion for valence. Saliency map analysis revealed that the model shifted its focus to the eyes and brow region. Interestingly, the AI also began treating the mask itself as a feature, interpreting its deformation as a proxy for hidden mouth movements.
Discussion and Conclusion
While masks initially hinder AI emotion recognition, this research demonstrates that lightweight models like FaceChannel are remarkably adaptable. By fine-tuning, the network learns to "look" at the eyes and even the mask's physical response to facial movement.
A primary limitation is the reliance on synthetic masks, which may not capture the full variety of real-world textures and fits. Future work will focus on continual learning systems capable of on-the-fly adaptation and testing in real-world social robotics environments where environmental factors like lighting and camera angles vary.
Summary updated