Abstract
Mask wearing has been considered an effective measure to prevent the spread of COVID-19 during the current pandemic. However, most advanced face recognition approaches are not adequate for masked face recognition, particularly in dealing with the issue of training through datasets covering only a limited number of images with ground-truth labels. In this work, we propose to learn from a large scale of web images and corresponding tags without any manual annotations, alongside limited fully annotated datasets. Inspired by the recent success of webly supervised learning in deep neural networks, we capitalize on readily-available web images with noisy annotations to learn a robust representation for masked faces. Beyond conventional spatial representation learning, we propose to leverage the power of the frequency domain to capture the local representative information of unoccluded facial parts. This approach learns robust feature embeddings derived from our feature fusion architecture to make joint and full use of information from both spatial and frequency domains. Experimental results on seven benchmarks show that the proposed approach significantly improves performance compared with other state-of-the-art methods.
Introduction: The Masked Face Recognition Challenge
Traditional facial recognition systems rely heavily on landmarks in the lower half of the face, such as the nose, mouth, and chin. The global adoption of mask-wearing during the COVID-19 pandemic caused these systems to fail significantly. The primary obstacle to improving these systems is data scarcity: while billions of standard face images exist, high-quality, manually labeled datasets of individuals wearing masks are small and expensive to produce.
This paper addresses two specific hurdles:
- The Data Gap: Utilizing the massive but "noisy" data available on the internet (images with tags but no verified labels) to improve recognition.
- Information Loss: Preventing neural networks from discarding useful details during downsampling, particularly when only the upper face is visible.
Methodology: Fusing Spatial and Frequency Domains
The researchers introduced a Spatial-Frequency Fusion Network integrated into a meta-learning "Relation Network" architecture. This framework moves beyond simple classification to learn the concept of identity by comparing "query" images against a "support" set.
Spatial-Frequency Fusion
Standard models look at raw pixels (the spatial domain). This framework adds a second branch that analyzes the frequency domain using Discrete Cosine Transform (DCT). This allows the model to capture fine-grained textures from unoccluded parts of the face, such as the eyes and forehead, which standard convolutions might overlook.
Two-Stage "Webly" Supervision
To handle the lack of clean data, the model uses a two-stage training process:
- Stage I: Pre-training on a small, clean dataset using Symmetric Cross Entropy to ensure robustness against initial label noise.
- Stage II: Adaptation using 5 million masked face images scraped from the web. To manage the noise inherent in web data (incorrect tags or poor crops), the authors employed Curriculum Learning, training on "easy," high-frequency faces before progressing to more difficult, rare examples.
Datasets and Diversity
The study utilizes a diverse mix of real-world, simulated, and web-scraped data. A key component is the custom "Web Image-set," which includes approximately 5 million images. To ensure the model generalizes across different populations, the researchers used race estimation to debias the web-scraped data.
Key Results and Insights
The proposed method consistently outperformed over a dozen state-of-the-art (SOTA) methods, including ArcFace and CosFace, across seven major benchmarks.
- Frequency Robustness: Ablation studies revealed that the frequency-domain branch (DCT-ResNet-50) often outperformed the spatial-domain branch. This suggests that when faces are occluded, texture information in the frequency domain provides a more reliable signal than pixel-based shape information.
- The Power of Fusion: The combined "Fusion" model was approximately 9% more accurate than a standard ResNet-50 alone on the SMFRD dataset.
- Generalization: In cross-database testing, the model maintained high performance (up to 99.61% accuracy), proving its ability to generalize to unseen individuals in real-world scenarios.
Conclusion
By combining webly supervised learning with spatial-frequency fusion, this framework provides a robust solution for masked face recognition. While the computational overhead of a dual-branch architecture is higher, the significant gains in accuracy and the ability to leverage massive amounts of unlabeled web data mark a major step forward in biometric security for an occluded world.
Summary updated