When Face Recognition Meets Occlusion: A New Benchmark

Baojin Huang, Zhongyuan Wang, Guangcheng Wang, Kui Jiang, Kangli Zeng, Zhen Han, Xin Tian, Yuhong Yang

TL;DR

Standard face recognition models fail dramatically when people wear masks or glasses—a problem highlighted by the COVID-19 pandemic. To fix this, researchers introduce Webface-OCC, a massive, realistically simulated occlusion dataset. By retraining existing models on this new benchmark, they achieved up to a 36% jump in accuracy on masked faces while barely sacrificing performance on normal, unoccluded faces.

What Problem Does This Paper Address?

Deep learning has made face recognition incredibly accurate under normal conditions, but performance plummets when faces are partially hidden. With mask-wearing becoming a global norm during the COVID-19 pandemic, this vulnerability became a critical failure point for real-world systems.

The core bottleneck is a lack of training data. Existing large-scale datasets mostly feature unoccluded faces. While some researchers have tried to synthesize occluded datasets, their methods have been overly simplistic—like slapping random black boxes over faces or pasting blunt, poorly aligned mask shapes onto images. These unnatural artifacts don't teach the neural network how to handle real-world occlusions, leading to poor generalization in the wild. This paper addresses the gap by creating a highly realistic, large-scale dataset of occluded faces to train more robust recognition models.

Methods

To build a better dataset, the authors moved away from random pixel-blocking and instead engineered a realistic Occlusion Simulation pipeline:

  1. Occluder Collection: They gathered a diverse set of real-world occlusion templates, including various types of glasses and masks (like surgical masks and N95s) with different colors and textures.
  2. Landmark Mapping: Using an open-source detector (RetinaFace), they extracted 68 facial landmarks from normal face images.
  3. Smart Alignment: They mapped the key points of the occluder templates to the facial landmarks. By dynamically adjusting the angle and size of the masks/glasses, they ensured the occluders fit the faces naturally, mimicking actual human wearing habits.
  4. Dataset Mixing: To ensure models trained on this data don't "forget" how to read normal faces, the final dataset retains a roughly 50/50 split of normal and occluded faces for each identity.

Datasets

  • Training Data:
  • CASIA-Webface: A standard baseline dataset of normal faces.
  • Webface-OCC (Newly Introduced): The authors' simulated dataset containing 804,704 images across 10,575 subjects, featuring a balanced mix of normal faces and realistically applied masks/glasses.
  • Testing/Evaluation Data:
  • Standard Benchmarks: LFW, CFP-FP, AgeDB-30.
  • Simulated Masked Benchmarks: LFW-mask, CFP-FP-mask, AgeDB-30-mask.
  • Real-World Masked Benchmark: RMFRD (Real-World Masked Face Recognition Dataset), which contains actual photos of people wearing masks in the wild.

Key Insights and Ideas

  • Realistic simulation is the secret sauce: Previous methods that used random black blocks or GAN-generated masks often destroyed underlying image details or created unnatural artifacts. By using geometric landmark alignment, the network learns to focus on the visible facial features (like the eyes and forehead) rather than getting confused by unnatural occlusion boundaries.
  • The "Sim-to-Real" gap is still a hurdle: While the models trained on Webface-OCC performed flawlessly on simulated test sets, their accuracy dropped when tested on real-world masked faces (RMFRD). The authors point out that real-world occlusions are incredibly diverse, and public figures (who make up most datasets) often wear masks specifically to disguise their identity, making the task inherently harder.
  • You can have your cake and eat it too: A common fear in machine learning is catastrophic forgetting—that training a model to recognize masked faces will ruin its ability to recognize normal ones. This paper proves that with a well-balanced dataset, you can massively boost occlusion robustness while only taking a negligible (~1%) hit to standard face recognition accuracy.

Key Results

The authors retrained several baseline models (like FaceNet and ArcFace) using their new Webface-OCC dataset. The quantitative gains are massive:

  • Simulated Masked Faces: Retrained ArcFace jumped from 60.86% to 97.08% accuracy on LFW-mask (a +36.22% improvement). Similar massive gains were seen on CFP-FP-mask (+29.14%) and AgeDB-30-mask (+27.04%).
  • Real-World Masked Faces: On the RMFRD dataset, the retrained ArcFace improved from 63.22% to 78.25% (+15.03%).
  • Normal Faces: The retrained ArcFace maintained a 99.01% accuracy on the standard LFW dataset, proving that the model didn't lose its baseline capabilities (compared to the original 99.53%).

Limitations and Future Work

  • Limitations: The authors acknowledge that accuracy still drops sharply when the False Positive Rate (FPR) is forced below 1e-3. This is a fundamental limitation of occlusion: when too many facial features are hidden, there simply isn't enough biometric data left to make a highly confident, low-error match. Additionally, the performance gap between simulated and real-world masked faces indicates that the simulation, while improved, doesn't capture all real-world edge cases.
  • Future Work: The authors plan to use this benchmark as a stepping stone to develop a "universal occlusion face recognition algorithm" that can handle any type of visual blockage seamlessly.

Summary updated

Abstract

The existing face recognition datasets usually lack occlusion samples, which hinders the development of face recognition. Especially during the COVID-19 coronavirus epidemic, wearing a mask has become an effective means of preventing the virus spread. Traditional CNN-based face recognition models trained on existing datasets are almost ineffective for heavy occlusion. To this end, we pioneer a simulated occlusion face recognition dataset. In particular, we first collect a variety of glasses and masks as occlusion, and randomly combine the occlusion attributes (occlusion objects, textures,and colors) to achieve a large number of more realistic occlusion types. We then cover them in the proper position of the face image with the normal occlusion habit. Furthermore, we reasonably combine original normal face images and occluded face images to form our final dataset, termed as Webface-OCC. It covers 804,704 face images of 10,575 subjects, with diverse occlusion types to ensure its diversity and stability. Extensive experiments on public datasets show that the ArcFace retrained by our dataset significantly outperforms the state-of-the-arts. Webface-OCC is available at https://github.com/Baojin-Huang/Webface-OCC.
BibTeX
@inproceedings{huang2021when,
  author = {Baojin Huang and Zhongyuan Wang and Guangcheng Wang and Kui Jiang and Kangli Zeng and Zhen Han and Xin Tian and Yuhong Yang},
  title = {{When Face Recognition Meets Occlusion: A New Benchmark}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  doi = {10.1109/icassp39728.2021.9413893},
}

Loading related content...

Figure