Occlusion Robust Face Recognition Based on Mask Learning with PairwiseDifferential Siamese Network

Lingxue Song, Dihong Gong, Zhifeng Li, Changsong Liu, Wei Liu

Abstract

Deep Convolutional Neural Networks (CNNs) have significantly advanced face recognition, yet they struggle with occlusions in variable facial regions. Humans naturally ignore occlusions to focus on visible areas; mimicking this, we propose a mask learning strategy to identify and discard corrupted feature elements. We introduce the Pairwise Differential Siamese Network (PDSN) to establish a mask dictionary by analyzing differences between occluded and occlusion-free face pairs. Each dictionary item, a Feature Discarding Mask (FDM), maps specific occluded areas to corrupted features. For recognition, we combine relevant FDMs to eliminate noise from the feature vector. Experiments demonstrate that our approach significantly outperforms state-of-the-art methods on both synthetic and real-world occluded datasets.

Introduction

While CNNs excel at general face recognition, they are fragile under partial occlusion. Models trained on clear faces often attempt to extract features from occluding objects (e.g., sunglasses), leading to corrupted feature vectors. Common data augmentation strategies often degrade performance on clean images. We propose a method that explicitly ignores occluded regions without compromising the recognition of unobstructed faces, leveraging the spatial consistency of deep features.

Proposed Methodology

The proposed framework operates in three distinct stages: mask generation learning, dictionary establishment, and robust recognition.

Overview of the proposed three-stage framework: mask generator learning, dictionary establishment, and feature-discarding recognition.

Stage I: Pairwise Differential Siamese Network (PDSN)

The core of our approach is the PDSN, designed to learn the correspondence between facial occlusions and feature corruption. It processes pairs of images—one clean ($x^i$) and one occluded ($x^j_i$) of the same person.

Architecture of the Pairwise Differential Siamese Network (PDSN) highlighting the differential signal and mask generator.

The network computes a Differential Signal by taking the absolute difference between the features of the clean and occluded faces. This signal serves as a direct indicator of which feature elements are affected by the occlusion. A Mask Generator then uses this signal to produce a mask $m^j_i$ that identifies corrupted elements.

Stage II: Establishing the Mask Dictionary

To handle arbitrary occlusions at test time, we pre-calculate masks for a 5x5 grid of facial blocks. For each block, we learn a binary Feature Discarding Mask (FDM). Binarization (setting values to 0 or 1) was found to be more effective than soft weighting, as it completely eliminates noise.

Stage III: Occlusion-Robust Recognition

During inference, an external occlusion detector (e.g., FCN-8s) identifies covered regions. The system retrieves the corresponding FDMs from the dictionary, combines them via a logical AND operation, and applies the resulting mask to the Trunk CNN's feature vector. This ensures only reliable, non-occluded features are used for matching.

Experimental Results

The model was trained on CASIA-WebFace using a ResNet50-based ArcFace "Trunk" CNN.

  • AR Dataset: On Protocol 2 (real-world occlusions), the PDSN achieved 98.19% accuracy for sunglasses and 98.33% for scarves, significantly surpassing the baseline Trunk CNN.
  • MegaFace: On the synthesized occluded set (MF1occ), accuracy improved to 56.34% from the baseline's 53.03%.
  • Clean Face Performance: Crucially, the model maintained 99.20% accuracy on the standard LFW benchmark, proving that robustness to occlusion does not come at the cost of general performance.

Conclusion

By explicitly discarding corrupted feature elements through a learned mask dictionary, our PDSN-based approach provides a robust solution for face recognition under occlusion. The use of a differential signal provides a strong supervision mechanism, allowing the network to precisely map spatial occlusions to feature-level noise.

Summary updated

Abstract

Deep Convolutional Neural Networks (CNNs) have been pushing the frontier of the face recognition research in the past years. However, existing general CNN face models generalize poorly to the scenario of occlusions on variable facial areas. Inspired by the fact that a human visual system explicitly ignores occlusions and only focuses on non-occluded facial areas, we propose a mask learning strategy to find and discard the corrupted feature elements for face recognition. A mask dictionary is firstly established by exploiting the differences between the top convoluted features of occluded and occlusion-free face pairs using an innovatively designed Pairwise Differential Siamese Network (PDSN). Each item of this dictionary captures the correspondence between occluded facial areas and corrupted feature elements, which is named Feature Discarding Mask (FDM). When dealing with a face image with random partial occlusions, we generate its FDM by combining relevant dictionary items and then multiply it with the original features to eliminate those corrupted feature elements. Comprehensive experiments on both synthesized and realistic occluded face datasets show that the proposed approach significantly outperforms the state-of-the-arts.
BibTeX
@inproceedings{song2019occlusion,
  author = {Lingxue Song and Dihong Gong and Zhifeng Li and Changsong Liu and Wei Liu},
  title = {{Occlusion Robust Face Recognition Based on Mask Learning with PairwiseDifferential Siamese Network}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00086},
}
Figure