MaskMTL: Attribute Prediction in Masked Facial Images with Deep Multitask Learning

Prerana Mukherjee, Vinay Kaushik, Ronak Gupta, Ritika Jha, Daneshwari Kankanwadi, Brejesh Lall

TL;DR

This paper introduces MaskMTL, a novel deep multi-task learning framework designed to predict facial attributes (age, gender, ethnicity) from images where faces are partially occluded by masks. By leveraging hierarchical parameter sharing and non-local attention, MaskMTL achieves competitive accuracy on masked faces, addressing a critical challenge in the post-pandemic era for applications like contactless authentication.

What Problem Does This Paper Address?

The COVID-19 pandemic has made face masks a ubiquitous part of daily life. This widespread adoption poses a significant challenge for facial recognition and attribute prediction systems, which are crucial for various applications, including secure access control, personal device unlocking, and even public health monitoring (e.g., identifying vulnerable groups). Traditional facial analysis methods often rely on clear, unoccluded facial features. When faces are masked, these systems struggle to extract reliable information, leading to decreased accuracy. While previous research has tackled general face occlusion, the specific and widespread occlusion caused by masks requires dedicated solutions. This paper aims to bridge this gap by developing a robust method for predicting key facial attributes from masked images.

Methods

The core of the proposed solution is a deep Multi-Task Learning (MTL) architecture called MaskMTL. The authors employ a hierarchical parameter sharing strategy, which is a hybrid approach combining benefits of hard and soft parameter sharing. In this setup, a shared encoder learns common representations for all tasks, while task-specific layers are introduced for each attribute. This allows for efficient feature learning and regularization, reducing the risk of overfitting.

Specifically, the architecture consists of:

  • A shared encoder: This part extracts general facial features.
  • Auxiliary tasks: Age estimation and gender recognition are treated as auxiliary tasks. These tasks help regularize the main task.
  • Main task: Ethnicity recognition is the primary focus.
  • Non-local attention: Introduced after the encoder, this mechanism helps the network focus on visible facial features, which is crucial when parts of the face are obscured by a mask. It computes responses by considering the weighted sum of features from all positions, effectively highlighting relevant facial regions.

The model is trained using a combination of L1 loss for age, categorical cross-entropy for ethnicity, and binary cross-entropy for gender.

Datasets

The primary dataset used is the UTKFace Dataset. This dataset is chosen for its diversity in age, gender, and ethnicity (White, Black, Asian, Indian, Others), and its inherent variations in illumination, pose, occlusion, and scale. To create masked facial images, the authors utilize the MaskTheFace tool. This tool generates masks with various textures and colors, applied to faces with different poses, effectively simulating real-world masked scenarios. The UTKFace dataset is split into training (Part I), testing (Part II), and validation (Part III) sets.

Key Insights and Ideas

  • Masked faces are a distinct domain: The authors acknowledge that simply augmenting masks onto existing datasets creates a new domain that requires specific model training, rather than relying solely on pre-trained models for unmasked faces.
  • Hierarchical parameter sharing is effective for attribute correlation: By structuring the MTL with auxiliary tasks (age, gender) influencing the main task (ethnicity), the model learns more robust and correlated features, leading to better performance.
  • Non-local attention is crucial for occluded faces: Focusing attention on visible facial regions significantly improves the model's ability to extract relevant information when masks obscure key features.
  • Lightweight architecture can achieve competitive results: The proposed encoder uses only 0.3M parameters, significantly fewer than larger models like ResNet-18 (11M parameters), yet achieves comparable or better performance on masked faces.

Key Results

  • Competitive accuracy on masked faces: The MaskMTL model achieves a gender accuracy of 0.8913, race accuracy of 0.7927, and an age L1-loss of 12.15 on the masked UTKFace dataset.
  • Outperforms standard MTL methods: When compared to various standard MTL approaches (e.g., Fixed MTL, GradNorm, Pareto MTL) on the masked dataset, MaskMTL demonstrates superior or comparable performance.
  • Improved performance with non-local attention: The inclusion of non-local attention is shown to further boost prediction accuracy.
  • Qualitative analysis reveals challenges: The paper highlights that age prediction can be challenging for very smooth faces (e.g., young children) or blurred images. Gender prediction struggles with side profiles or ambiguous features. Ethnicity prediction can be confused by similar-looking races or poor image contrast.

Limitations and Future Work

  • Low recall for "Others" ethnicity: The authors note that the model has a low recall for the "Others" ethnicity class, suggesting that the presence of similar attributes across different ethnicities makes it difficult to distinguish.
  • Future work: The authors intend to explore learning other facial attributes beyond age, gender, and ethnicity. They also suggest that pre-training the model on a larger, unmasked face dataset could further improve performance.

Summary updated

Abstract

Predicting attributes in the landmark free facial images is itself a challenging task which gets further complicated when the face gets occluded due to the usage of masks. Smart access control gates which utilize identity verification or the secure login to personal electronic gadgets may utilize face as a biometric trait. Particularly, the Covid-19 pandemic increasingly validates the essentiality of hygienic and contactless identity verification. In such cases, the usage of masks become more inevitable and performing attribute prediction helps in segregating the target vulnerable groups from community spread or ensuring social distancing for them in a collaborative environment. We create a masked face dataset by efficiently overlaying masks of different shape, size and textures to effectively model variability generated by wearing mask. This paper presents a deep Multi-Task Learning (MTL) approach to jointly estimate various heterogeneous attributes from a single masked facial image. Experimental results on benchmark face attribute UTKFace dataset demonstrate that the proposed approach supersedes in performance to other competing techniques. The source code is available at https://github.com/ritikajha/Attribute-prediction-in-masked-facial-images-with-deep-multitask-learning
BibTeX
@inproceedings{mukherjee2021maskmtl,
  author = {Prerana Mukherjee and Vinay Kaushik and Ronak Gupta and Ritika Jha and Daneshwari Kankanwadi and Brejesh Lall},
  title = {{MaskMTL: Attribute Prediction in Masked Facial Images with Deep Multitask Learning}},
  booktitle = {Premi},
  year = {2021},
  doi = {10.1007/978-3-031-12700-7\_42},
}
Figure