Complete Face Recovery GAN: Unsupervised Joint Face Rotation and De-Occlusion from a Single-View Image

Yeong-Joon Ju, Gun-Hee Lee, Jung-Ho Hong, Seong-Whan LeeWACV 2022 · OpenReview

Abstract

Although various face-related tasks have significantly advanced in recent years, occlusion and extreme pose still impede the achievement of higher performance. Existing face rotation or de-occlusion methods only have emphasized the aspect of each problem. In addition, the lack of high-quality paired data remains an obstacle for both methods. In this work, we present a self-supervision strategy called Swap-R&R to overcome the lack of ground-truth in a fully unsupervised manner for joint face rotation and de-occlusion. To generate an input pair for self-supervision, we transfer the occlusion from a face in an image to an estimated 3D face and create a damaged face image, as if rotated from a different pose by rotating twice with the roughly de-occluded face. Furthermore, we propose Complete Face Recovery GAN (CFR-GAN) to restore the collapsed textures and disappeared occlusion areas by leveraging the structural and textural differences between two rendered images. Unlike previous works, which have selected occlusion-free images to obtain ground-truths, our approach does not require human intervention and paired data. We show that our proposed method can generate a de-occluded frontal face image from an occluded profile face image. Moreover, extensive experiments demonstrate that our approach can boost the performance of facial recognition and facial expression recognition.

Introduction

In computer vision, two of the most significant hurdles for facial recognition and expression analysis are extreme poses (such as sharp profile views) and occlusions (objects like sunglasses, hands, or masks blocking the face). While researchers have tackled these problems individually, addressing both simultaneously remains a formidable challenge.

The primary bottleneck is the lack of high-quality paired data. Training a standard deep learning model typically requires thousands of image pairs showing the same individual in identical poses, both with and without occlusions. Since such datasets do not exist naturally, most models rely on artificial occlusions that lack realism, leading to poor performance on real-world images. CFR-GAN addresses this by performing face rotation and de-occlusion simultaneously from a single image without requiring any paired training data.

Methodology: The CFR-GAN Framework

The Complete Face Recovery GAN (CFR-GAN) operates in a fully unsupervised manner by leveraging 3D Morphable Models (3DMM) and a novel self-supervision strategy. The framework consists of two primary components:

1. Occlusion-Robust 3D Reconstruction

The system first fits a 3DMM to the input image. Because 3DMMs are statistical models of "average" faces, they are naturally "clean" and struggle to represent non-facial occlusions. The authors utilize this limitation as a feature, using the clean 3D model as a structural guide for the underlying facial geometry hidden beneath occlusions.

2. Swap-R&R (Swap, Rotate, and Render)

To overcome the lack of ground truth, the authors introduced the Swap-R&R strategy. They generate two 3D faces: a "clean" face from 3DMM parameters and a "projected" face that maps the original occluded texture onto the 3D shape. By swapping occluded regions from the projected face onto the clean face and performing a double rotation-and-render process, they create a "damaged" input pair. The GAN then learns to reconstruct the original image from this synthetic version, effectively training itself to remove occlusions and restore textures during rotation.

The CFR-GAN pipeline utilizes the Swap-R&R strategy for self-supervised training, leveraging 3D structural guides to recover occluded facial regions.

The architecture is further enhanced with an Occlusion Parsing Path (O path), which uses 3D-guided attention to identify "real face" versus "occlusion" pixels, allowing the generator to focus restorative power where it is most needed.

Datasets and Evaluation

The model was trained and evaluated on several high-standard benchmarks to ensure robustness across various tasks:

  • Training: CelebA-HQ, CelebA, and FFHQ.
  • Face Recognition: LFW and the challenging IJB-A, IJB-B, and IJB-C datasets, which are rich in profile and occluded views.
  • Expression Recognition: RAF-DB (Real-world Affective Faces Database).
  • 3D Alignment: AFLW2000-3D.

Key Results and Insights

CFR-GAN demonstrates that joint optimization of rotation and de-occlusion is superior to treating them as separate steps. By doing both at once, the model uses 3D structure to maintain identity and spatial consistency more effectively than 2D-only approaches.

  • Face Recognition Boost: On the difficult IJB-C dataset, using CFR-GAN as a pre-processor boosted the True Acceptance Rate (TAR) to 86.46% (at FAR=0.001), a significant improvement over the 79.39% baseline.
  • Expression Recognition: The model achieved 85.48% accuracy on RAF-DB, outperforming specialized models like GACNN.
  • Qualitative Performance: The model successfully removes complex occlusions, such as hands and microphones, and rotates faces up to 90 degrees while maintaining identity and producing photorealistic skin textures.
Qualitative results demonstrating the model's ability to frontalize and de-occlude faces across various challenging real-world scenarios.

Limitations and Future Work

While highly effective, the model's output quality is dependent on the initial 3D face reconstruction. If the 3DMM fit is significantly inaccurate due to extreme lighting or highly unusual face shapes, the recovery will be flawed. Additionally, while the model handles common occlusions well, it may produce artifacts when encountering very unusual objects not represented during training. Future work may focus on improving the robustness of the 3D fitting process to further enhance the GAN's restorative capabilities.

Summary updated

Abstract

Although various face-related tasks have significantly advanced in recent years, occlusion and extreme pose still impede the achievement of higher performance. Existing face rotation or de-occlusion methods only have emphasized the aspect of each problem. In addition, the lack of high-quality paired data remains an obstacle for both methods. In this work, we present a self-supervision strategy called Swap-R&R to overcome the lack of ground-truth in a fully unsupervised manner for joint face rotation and de-occlusion. To generate an input pair for self-supervision, we transfer the occlusion from a face in an image to an estimated 3D face and create a damaged face image, as if rotated from a different pose by rotating twice with the roughly de-occluded face. Furthermore, we propose Complete Face Recovery GAN (CFR-GAN) to restore the collapsed textures and disappeared occlusion areas by leveraging the structural and textural differences between two rendered images. Unlike previous works, which have selected occlusion-free images to obtain ground-truths, our approach does not require human intervention and paired data. We show that our proposed method can generate a de-occluded frontal face image from an occluded profile face image. Moreover, extensive experiments demonstrate that our approach can boost the performance of facial recognition and facial expression recognition. The code is publicly available <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
BibTeX
@inproceedings{ju2022complete,
  author = {Yeong-Joon Ju and Gun-Hee Lee and Jung-Ho Hong and Seong-Whan Lee},
  title = {{Complete Face Recovery GAN: Unsupervised Joint Face Rotation and De-Occlusion from a Single-View Image}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
  url = {https://openreview.net/forum?id=BQOWJjCv3m},
}
Figure