Exemplar Guided Face Image Super-Resolution without Facial Landmarks

Berk Dogan, Shuhang Gu, Radu Timofte

Abstract

Nowadays, due to the ubiquitous visual media there are vast amounts of already available high-resolution (HR) face images. Therefore, for super-resolving a given very low-resolution (LR) face image of a person it is very likely to find another HR face image of the same person which can be used to guide the process. In this paper, we propose a convolutional neural network (CNN)-based solution, namely GWAInet, which applies super-resolution (SR) by a factor 8x on face images guided by another unconstrained HR face image of the same person with possible differences in age, expression, pose or size. GWAInet is trained in an adversarial generative manner to produce the desired high quality perceptual image results. The utilization of the HR guiding image is realized via the use of a warper subnetwork that aligns its contents to the input image and the use of a feature fusion chain for the extracted features from the warped guiding image and the input image. In training, the identity loss further helps in preserving the identity related features by minimizing the distance between the embedding vectors of SR and HR ground truth images. Contrary to the current state-of-the-art in face super-resolution, our method does not require facial landmark points for its training, which helps its robustness and allows it to produce fine details also for the surrounding face region in a uniform manner. Our method GWAInet produces photo-realistic images in upscaling factor 8x and outperforms state-of-the-art in quantitative terms and perceptual quality.

Introduction

Super-resolving a face by a factor of 8x—for instance, transforming a $16 \times 16$ pixel patch into a clear portrait—is a fundamentally ill-posed problem. While standard methods often yield blurry outputs, Generative Adversarial Networks (GANs) can hallucinate sharp details. However, these hallucinations frequently fail to preserve the specific identity of the individual.

Previous state-of-the-art approaches attempted to mitigate this by using a high-resolution "exemplar" image of the same person as a reference. These methods typically relied on facial landmark detection (eyes, nose, mouth) to align the reference image with the low-resolution input. This creates a "chicken-and-egg" dilemma: accurate landmark detection is nearly impossible on the extremely blurry images that the system is trying to enhance. When landmark detection fails, the resulting reconstructions are often distorted or only partially sharp.

The GWAInet Architecture

To address these challenges, we propose GWAInet, an end-to-end pipeline that avoids landmark dependency. The architecture consists of four specialized subnetworks:

  1. Warper (Wnet): Utilizing a Spatial Transformer Network approach, Wnet calculates a "flow field" to automatically warp and align the high-resolution guiding image to the pose and expression of the low-resolution input.
  2. Generator (Gnet): The core engine comprising two branches: SRnet for processing the low-resolution input and GFEnet for extracting features from the warped guide. These features are fused to inject high-frequency details like hair texture and iris patterns.
  3. Critic (Cnet): A discriminator based on the WGAN-GP framework that ensures the production of photo-realistic textures.
  4. Identity Encoder (Inet): A Siamese network that calculates an Identity Loss, ensuring the super-resolved face maintains the "fingerprint" of the original person.
The GWAInet architecture, featuring the Warper (Wnet), Generator (Gnet), Critic (Cnet), and Identity Encoder (Inet).

Experimental Results

The model was evaluated on the CelebA dataset for initial development and a subset of VggFace2 & CASIA-WebFace for benchmarking against pairs of guiding and ground-truth images.

Quantitative Performance

GWAInet demonstrated significant quantitative improvements over previous methods. On the VggFace2 dataset, it outperformed the previous state-of-the-art (GFRNet) by 1.47 dB in Peak Signal-to-Noise Ratio (PSNR).

Qualitative Realism

Visually, GWAInet produces sharper results with fewer artifacts. By treating the face as a whole rather than focusing on landmarks, the model achieves uniform detail reconstruction across the entire image, including often-neglected areas like the forehead and hair.

Qualitative comparison of super-resolution results, demonstrating the model's ability to reconstruct fine facial details and textures.

Discussion and Conclusion

A key insight of this work is the robustness gained by "ditching" landmarks. The model successfully "borrows" textures from the guiding image even when there are discrepancies in age or expression. However, limitations remain; the model is sensitive to the identity of the guide—using an incorrect person can lead to identity shifting in the output. Additionally, extreme pose discrepancies (e.g., profile vs. frontal) continue to be a challenge.

In conclusion, GWAInet provides a robust, landmark-free solution for 8x face super-resolution, achieving state-of-the-art performance in both quantitative metrics and perceptual quality.

Summary updated

Abstract

Nowadays, due to the ubiquitous visual media there are vast amounts of already available high-resolution (HR) face images. Therefore, for super-resolving a given very low-resolution (LR) face image of a person it is very likely to find another HR face image of the same person which can be used to guide the process. In this paper, we propose a convolutional neural network (CNN)-based solution, namely GWAInet, which applies super-resolution (SR) by a factor 8x on face images guided by another unconstrained HR face image of the same person with possible differences in age, expression, pose or size. GWAInet is trained in an adversarial generative manner to produce the desired high quality perceptual image results. The utilization of the HR guiding image is realized via the use of a warper subnetwork that aligns its contents to the input image and the use of a feature fusion chain for the extracted features from the warped guiding image and the input image. In training, the identity loss further helps in preserving the identity related features by minimizing the distance between the embedding vectors of SR and HR ground truth images. Contrary to the current state-of-the-art in face super-resolution, our method does not require facial landmark points for its training, which helps its robustness and allows it to produce fine details also for the surrounding face region in a uniform manner. Our method GWAInet produces photo-realistic images in upscaling factor 8x and outperforms state-of-the-art in quantitative terms and perceptual quality.
BibTeX
@inproceedings{dogan2019exemplar,
  author = {Berk Dogan and Shuhang Gu and Radu Timofte},
  title = {{Exemplar Guided Face Image Super-Resolution without Facial Landmarks}},
  booktitle = {Workshops @ CVPR},
  year = {2019},
  url = {https://openreview.net/forum?id=veI5unILrsem},
}
Figure