Medical Face Masks and Emotion Recognition from the Body: Insights from a Deep Learning Perspective

Nikolaos Kegkeroglou, Panagiotis Paraskevas Filntisis, Petros Maragos

Abstract

The COVID-19 pandemic has undoubtedly changed the standards and affected all aspects of our lives, especially social communication. It has forced people to extensively wear medical face masks, in order to prevent transmission. This face occlusion can strongly irritate emotional reading from the face and urges us to incorporate the whole body as an emotional cue. In this paper, we conduct insightful studies about the effect of face occlusion on emotion recognition performance, and showcase the superiority of full body input over the plain masked face. We utilize a deep learning model based on the Temporal Segment Network framework, and aspire to fully overcome the face mask consequences. Although facial and bodily features can be learned from a single input, this may lead to irrelevant information confusion. By processing those features separately and fusing their prediction scores, we are more effectively taking advantage of both modalities. This framework also naturally supports temporal modeling, by mingling information among neighboring frames. In combination, these techniques form an effective system capable of tackling emotion recognition difficulties, caused by safety protocols applied in crucial areas.

Introduction

Historically, automated emotion recognition has relied heavily on facial expressions, specifically focusing on the mouth and jawline. However, the global adoption of medical face masks has introduced a significant occlusion problem for computer vision systems. This challenge is particularly acute in fields like Child-Robot Interaction (CRI), where a robot tutor must accurately gauge a child's emotional state—such as frustration or confusion—to provide appropriate support. Given that children often move dynamically or turn away from cameras, the addition of a mask makes traditional facial-only models nearly ineffective. This research investigates how leveraging full-body cues (posture, hand gestures, and shoulder shrugs) can compensate for the loss of facial information.

Methodology

The proposed system utilizes a Temporal Segment Network (TSN) framework with a ResNet-50 backbone. To address the challenges of masked emotion recognition, several key engineering strategies were implemented:

Synthetic Masking

To study the impact of masks on existing datasets, the authors developed a tool using MediaPipe Face Mesh. This tool programmatically applies realistic medical masks to video frames, adjusting for head orientation and geometry to simulate real-world occlusion.

Figure 3: Steps for applying synthetic masks to video frames using mesh tracking.

Modality Fusion and Temporal Modeling

To avoid "information confusion" where the network might struggle to distinguish between facial and bodily signals, the researchers processed the face and the body (with the face area blacked out) in two separate streams. They then employed Late Fusion, averaging the prediction scores from both streams for the final result. Furthermore, the Temporal Shift Module (TSM) was integrated to allow the network to understand the dynamics of gestures over time without increasing computational costs.

Dataset Analysis

The study utilized the EmoReact dataset, which consists of 1,102 videos of 63 children (ages 4–14). The dataset is a binary multi-label problem, where clips are annotated with eight emotions: Curiosity, Uncertainty, Excitement, Happiness, Surprise, Disgust, Fear, and Frustration.

Figure 1: Distribution of emotion labels in the EmoReact dataset, highlighting significant class imbalance.

As shown in the distribution, the dataset suffers from significant imbalance, particularly for emotions like "Fear" and "Frustration," which presents a challenge for model generalization.

Key Insights and Results

  • Robustness of Body Cues: While masking causes a 3–4% performance drop in facial-only models, the "Full Body" model's performance decreased by less than 0.4%, confirming that bodily expressions are highly robust to facial occlusion.
  • Performance Recovery: The best-performing model (TSM + Fusion) achieved a 0.768 ROC AUC. This nearly matches the best possible result on unmasked faces (0.769), suggesting that a body-centric approach can almost entirely mitigate the negative impact of masks.
  • Emotion-Specific Performance: The body was found to be particularly effective for recognizing Fear. Conversely, Excitement and Frustration remained more dependent on facial cues, even when masked.
  • Visual Attention: Grad-CAM visualizations revealed that the model focuses on specific bodily regions for different emotions: hands for "Happiness," wide-open arms for "Excitement," and shrugged shoulders for "Curiosity."

Conclusion and Future Work

This research demonstrates that body language is a powerful and necessary modality for emotion recognition in the age of masks. While synthetic masking and modality fusion have proven effective, future work should address dataset imbalances and explore the use of 3D skeletal data to further improve robustness in complex, real-world Child-Robot Interaction scenarios.

Summary updated

Abstract

The COVID-19 pandemic has undoubtedly changed the standards and affected all aspects of our lives, especially social communication. It has forced people to extensively wear medical face masks, in order to prevent transmission. This face occlusion can strongly irritate emotional reading from the face and urges us to incorporate the whole body as an emotional cue. In this paper, we conduct insightful studies about the effect of face occlusion on emotion recognition performance, and showcase the superiority of full body input over the plain masked face. We utilize a deep learning model based on the Temporal Segment Network framework, and aspire to fully overcome the face mask consequences. Although facial and bodily features can be learned from a single input, this may lead to irrelevant information confusion. By processing those features separately and fusing their prediction scores, we are more effectively taking advantage of both modalities. This framework also naturally supports temporal modeling, by mingling information among neighboring frames. In combination, these techniques form an effective system capable of tackling emotion recognition difficulties, caused by safety protocols applied in crucial areas.
BibTeX
@inproceedings{kegkeroglou2023medical,
  author = {Nikolaos Kegkeroglou and Panagiotis Paraskevas Filntisis and Petros Maragos},
  title = {{Medical Face Masks and Emotion Recognition from the Body: Insights from a Deep Learning Perspective}},
  booktitle = {PETRA},
  year = {2023},
  doi = {10.1145/3594806.3594829},
}
Figure