Occlusion Robust Face Recognition Based on Mask Learning With Pairwise Differential Siamese Network

Lingxue Song, Dihong Gong, Zhifeng Li, Changsong Liu, Wei Liu

Abstract

Deep Convolutional Neural Networks (CNNs) have significantly advanced face recognition, yet they remain vulnerable to facial occlusions such as sunglasses or masks. This paper presents a novel approach using a Pairwise Differential Siamese Network (PDSN) to identify and discard corrupted feature elements. By establishing a "Mask Dictionary" based on facial regions, the system can dynamically mask noisy features during recognition, maintaining high accuracy on both occluded and clean faces.

Introduction

While deep CNNs excel at extracting high-level features, they often operate as "black boxes" regarding spatial occlusions. When a portion of a face is obscured, standard networks still attempt to extract features from that region, leading to "corrupted" feature vectors filled with noise. Traditional methods like HOG or SIFT allowed for direct spatial-to-feature mapping, but in deep learning, a single neuron's receptive field may cover a large, ambiguous area. Simply augmenting training data with occluded images often leads to a performance trade-off on clean images. This research proposes a method to explicitly identify which deep feature elements are affected by specific occluded facial areas and discard them during the recognition process.

Methodology

The proposed framework consists of three primary stages: learning mask generators, establishing a mask dictionary, and performing occlusion-robust recognition.

Overview of the proposed three-stage framework for occlusion-robust face recognition.

Pairwise Differential Siamese Network (PDSN)

The core innovation is the PDSN, designed to learn the relationship between occluded facial blocks and corrupted feature elements. The network processes pairs of images: a clean face ($x^i$) and an occluded version of the same face ($x^i_j$). By calculating the differential signal—the absolute difference between the feature maps $f(x^i)$ and $f(x^i_j)$—the network isolates the changes specifically caused by the occlusion. A "Mask Generator" module then learns to map these differences to a mask $m^i_j$.

Detailed architecture of the Pairwise Differential Siamese Network (PDSN) used to generate masks from differential signals.

Mask Dictionary and Feature Discarding

The face is partitioned into a $5 \times 5$ grid. For each grid block, the PDSN learns a specific mask identifying the feature elements most susceptible to corruption when that block is occluded. These masks are binarized into Feature Discarding Masks (FDMs) and stored in a dictionary. This approach favors discarding corrupted information over attempting to reconstruct it (e.g., via GANs), as reconstruction often introduces further artifacts.

Recognition Pipeline

During inference, an external segmentation network first identifies occluded regions. The system retrieves the corresponding FDMs from the dictionary, combines them using a logical "AND" operation, and applies the resulting mask to the face's feature vector. This "zeros out" corrupted components before final classification or comparison.

Experimental Results

The method was evaluated across several benchmarks, demonstrating significant improvements over baseline models.

  • AR Dataset: On Protocol 2, the proposed method achieved 98.19% accuracy for sunglasses and 98.33% for scarves, compared to the baseline Trunk CNN's 95.14% and 96.53%, respectively.
  • LFW Benchmark: The model maintained a high accuracy of 99.20% on clean faces, avoiding the performance degradation typically seen with standard occlusion-augmentation strategies (which dropped to 98.68%).
  • Large-Scale Scalability: On the MegaFace Challenge 1 with synthesized occlusions (MF1occ), the method reached 56.34% accuracy, outperforming the baseline of 53.03%.

Conclusion

By leveraging the differential signal between clean and occluded face pairs, the PDSN effectively identifies corrupted deep features. The use of a binary Mask Dictionary allows for a robust recognition system that ignores noise without sacrificing performance on unoccluded images. Future work may involve refining the fixed-grid approach and extending this masking philosophy to other forms of image degradation like motion blur.

Summary updated

Abstract

Deep Convolutional Neural Networks (CNNs) have been pushing the frontier of the face recognition research in the past years. However, existing general CNN face models generalize poorly to the scenario of occlusions on variable facial areas. Inspired by the fact that a human visual system explicitly ignores occlusions and only focuses on non-occluded facial areas, we propose a mask learning strategy to find and discard the corrupted feature elements for face recognition. A mask dictionary is firstly established by exploiting the differences between the top convoluted features of occluded and occlusion-free face pairs using an innovatively designed Pairwise Differential Siamese Network (PDSN). Each item of this dictionary captures the correspondence between occluded facial areas and corrupted feature elements, which is named Feature Discarding Mask (FDM). When dealing with a face image with random partial occlusions, we generate its FDM by combining relevant dictionary items and then multiply it with the original features to eliminate those corrupted feature elements. Comprehensive experiments on both synthesized and realistic occluded face datasets show that the proposed approach significantly outperforms the state-of-the-arts.
BibTeX
@inproceedings{song2019occlusion,
  author = {Lingxue Song and Dihong Gong and Zhifeng Li and Changsong Liu and Wei Liu},
  title = {{Occlusion Robust Face Recognition Based on Mask Learning With Pairwise Differential Siamese Network}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00086},
}
Figure