A Unified Framework for Masked and Mask-Free Face Recognition Via Feature Rectification

Shaozhe Hao, Chaofeng Chen, Zhenfang Chen, Kwan-Yee K. Wong

Introduction

Face recognition technology has reached high levels of maturity under ideal conditions. However, the global COVID-19 pandemic introduced a significant challenge: widespread facial occlusion due to masks. Standard models often fail when key facial features like the nose and mouth are hidden. Conversely, models fine-tuned specifically on masked data frequently suffer from "catastrophic forgetting," where their performance on unmasked, clean faces degrades significantly. This paper presents the Face Feature Rectification Network (FFR-Net), a unified framework designed to handle both masked and mask-free faces within a single, consistent feature space.

Methodology: The FFR-Net Architecture

The FFR-Net approach moves away from visual pixel restoration (e.g., using GANs), which often fails to preserve mathematical identity information. Instead, it focuses on feature rectification.

The FFR-Net architecture featuring spatial and channel rectification blocks.

The framework utilizes a pre-trained SENet50 encoder to extract initial features. These features are then processed by two parallel Rectification Blocks (RecBlocks):

  • Spatial Rectification (SpcRec): Learns to identify and suppress corrupted spatial regions.
  • Channel Rectification (ChnRec): Adjusts feature channels to emphasize identity-relevant information.

To enhance robustness, the model incorporates Self-Similarity Inputs, calculating cosine similarity matrices to infer missing structural patterns from visible regions. Additionally, Flipped Fusion is employed within the channel rectification block to enforce spatial symmetry, ensuring the model leverages information from both sides of the face. For deployment, the authors suggest FFR-Net+, which fuses similarity scores from the rectified model with the original backbone to maintain peak performance on clean faces.

Key Insights: Achieving a Unified Feature Space

The fundamental innovation of FFR-Net is its ability to project corrupted features into a "rectified" latent space that aligns with clean features. In a standard recognition model, masked and unmasked embeddings of the same person typically form separate, distant clusters. FFR-Net overcomes this by learning a mask-resistant representation.

Visualization of the rectified feature space where masked and unmasked embeddings of the same identity align tightly.

As shown in the feature space visualization, the rectified embeddings for various identities (represented by different colors) show that masked samples (triangles) and unmasked samples (circles) of the same individual cluster together. This demonstrates the model's success in creating an identity-consistent representation regardless of occlusion.

Results and Performance

FFR-Net was evaluated on several benchmarks, including MegaFace (MF1) and real-world masked datasets (RMFD).

  • Masked Accuracy: On the MF1 dataset, FFR-Net recovered 25.06% of the accuracy lost by the baseline model when faces were masked.
  • Clean Face Stability: The model maintained high performance on unmasked faces, with only a 6.48% drop compared to the baseline, which can be further mitigated using the FFR-Net+ score fusion technique.
  • Real-World Generalization: The model achieved 93.08% accuracy on real-world masked data, outperforming the baseline by over 3%.
  • Efficiency: The rectification blocks add only 17.9% computational overhead, making the framework suitable for real-time applications.

Conclusion

FFR-Net provides a robust, unified solution for face recognition in the presence of masks. By focusing on feature-level rectification rather than pixel-level reconstruction, the framework achieves state-of-the-art performance on mixed datasets. Future work will likely explore training on larger real-world masked datasets to further close the performance gap on clean faces without requiring score fusion.

Summary updated

BibTeX
@inproceedings{hao2022unified,
  author = {Shaozhe Hao and Chaofeng Chen and Zhenfang Chen and Kwan-Yee K. Wong},
  title = {{A Unified Framework for Masked and Mask-Free Face Recognition Via Feature Rectification}},
  booktitle = {ICIP},
  year = {2022},
  url = {https://openreview.net/forum?id=ygSAnJJhMj4},
}
Figure