Face Super-Resolution Guided by 3D Facial Priors

Xiaobin Hu, Wenqi Ren, John LaMaster, Xiaochun Cao, Xiaoming Li, Zechao Li, Bjoern H. Menze, Wei Liu

Introduction

Face Super-Resolution (FSR), often referred to as face hallucination, is a specialized task in computer vision aimed at recovering high-resolution (HR) facial images from low-resolution (LR) inputs. Unlike general image super-resolution, FSR requires the preservation of complex structural details and identity-specific features. While traditional methods have increasingly relied on 2D priors—such as facial landmarks or parsing maps—these approaches often falter when faced with large pose variations or extreme downsampling factors. This paper presents a novel framework that integrates 3D facial priors to provide robust structural guidance, significantly improving reconstruction quality in challenging "in-the-wild" scenarios.

The Limitations of 2D Priors

Existing FSR techniques typically utilize 2D geometric information to guide the hallucination process. However, 2D landmarks are highly sensitive to head orientation; as a face turns, landmarks may become occluded or overlap, leading to distorted reconstructions. Furthermore, 2D maps lack depth and shading information, which are critical for maintaining realistic facial volume and lighting consistency. When the input image is as small as 16x16 pixels, 2D-based methods often produce "ghosting" artifacts or "waxy" textures that fail to capture the subject's true identity.

Low-resolution input (16x16 pixels) where structural details are severely degraded.

Methodology: 3D-Guided Reconstruction

The proposed architecture addresses these issues through a two-branch system: a 3D Rendering Branch and a Spatial Attention Module (SAM3D).

3D Rendering Branch

The core innovation lies in the use of a 3D Morphable Model (3DMM). Using a ResNet-50 backbone, the network regresses coefficients for identity, expression, texture, illumination, and pose from the LR input. These coefficients are then used to render a sharp 3D face. This rendered image serves as a "blueprint," providing the super-resolution branch with a consistent 3D skeleton that is inherently pose-invariant. To ensure the model focuses on relevant facial features, a skin-aware loss function is employed during training to mask out non-facial background noise.

Spatial Attention Module (SAM3D)

To effectively merge the 3D prior with the LR features, the authors introduced the SAM3D. This module utilizes Spatial Feature Transform (SFT) layers to adaptively scale and shift feature maps based on the 3D guidance. By incorporating Residual Channel Attention Blocks (RCAB), the network can prioritize informative regions—such as the eyes, nose, and mouth—even when the initial input is heavily blurred. This hierarchical approach allows the model to leverage both low-level edge information and high-level identity cues.

Experimental Results

The framework was evaluated on the CelebA dataset for general performance and the Menpo dataset for robustness against large pose variations.

Quantitative and Qualitative Performance

On the CelebA 8x scale task, the proposed method achieved a Peak Signal-to-Noise Ratio (PSNR) of 25.39 dB, outperforming established models like RDN (23.69 dB) and FSRNet (22.62 dB). The advantage was even more pronounced on the Menpo dataset, where the 3D-guided approach surpassed 2D-based methods by nearly 2 dB in large-pose scenarios.

Reconstructed high-resolution face using the proposed SAM3D framework, showing clear definition in the eyes and mouth.

Qualitatively, the results are markedly superior. While competing methods often produce blurry or distorted features, the 3D-guided model generates sharp, well-defined textures. The integration of 3D priors ensures that the reconstructed face maintains structural integrity regardless of the subject's orientation.

Discussion and Future Work

One of the most significant findings is the "plug-and-play" flexibility of the 3D prior. The authors demonstrated that adding their 3D guidance to simpler networks like SRCNN or VDSR resulted in substantial quality improvements, suggesting that structural knowledge is a more critical factor than network depth alone in FSR.

However, the method is not without limitations. It is constrained by the representational power of the 3DMM; unique facial features or extreme accessories not covered by the model may not be perfectly reconstructed. Additionally, the computational overhead of the dual-branch architecture is higher than that of single-branch CNNs. Future research will focus on improving robustness against extreme occlusions and optimizing the model for real-time applications.

Conclusion

By leveraging 3D facial priors, this research sets a new benchmark for face super-resolution. The transition from 2D landmarks to 3D morphable models provides the necessary geometric consistency to handle large poses and high magnification factors, paving the way for more reliable face hallucination in real-world surveillance and forensic applications.

Summary updated

BibTeX
@inproceedings{hu2020face,
  author = {Xiaobin Hu and Wenqi Ren and John LaMaster and Xiaochun Cao and Xiaoming Li and Zechao Li and Bjoern H. Menze and Wei Liu},
  title = {{Face Super-Resolution Guided by 3D Facial Priors}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  doi = {10.1007/978-3-030-58548-8\_44},
}
Figure