Semantic Encoder Guided Generative Adversarial Face Ultra-Resolution Network

Xiang Wang, Yimin Yang, Qixiang Pang, Xiao Lu, Yu Liu, Shan DuarXiv 2022 · OpenReview

TL;DR

This paper introduces SEGA-FURN, a novel face super-resolution network that leverages a semantic encoder to guide the generative adversarial process. By incorporating semantic information alongside visual data, SEGA-FURN achieves state-of-the-art results in generating high-resolution face images from extremely low-resolution inputs, even with unaligned faces.

What Problem Does This Paper Address?

Face Super-Resolution (FSR) is a challenging task in computer vision, aiming to reconstruct high-resolution (HR) face images from low-resolution (LR) counterparts. Existing methods often struggle with complex facial expressions, lost facial information in LR images, and the need for precise facial alignment or prior knowledge like landmarks. Many deep learning-based FSR methods, while improving results, can produce overly smooth images due to reconstruction losses (L1/L2). Generative Adversarial Networks (GANs) offer a path to more perceptually realistic results, but they are prone to training instability, gradient vanishing, and model collapse. Furthermore, prior knowledge-based methods, while effective, demand significant computational resources and are not suitable for real-time applications. This paper addresses these limitations by proposing a GAN-based FSR method that is robust to unaligned faces, avoids explicit prior knowledge, and enhances training stability for superior FSR.

Methods

The proposed SEGA-FURN is a GAN-based framework with four key components:

  • Semantic Encoder (E): This novel component extracts "embedded semantics" from both HR and LR face images. Unlike previous methods that solely rely on visual information, this encoder captures facial attributes (shape, gender) and spatial relationships, which are then used to guide the discriminator. It utilizes pre-trained VGG19 networks, adapted for different input dimensions.
  • Generator (G): The generator employs a hierarchical architecture called Residual in Internal Dense Block (RIDB). This design, building upon previous work, effectively extracts and fuses hierarchical features, enhancing the generator's capability. It consists of a Shallow Feature Module (SFM), a Multi-level Dense Block Module (MDBM) composed of DNBs (which contain RIDBs), and an Upsampling Module (UM).
  • Joint Discriminator (D): This discriminator is designed to evaluate tuples containing both visual information (HR/SR images) and embedded semantics. It comprises two sub-nets: ESLDSN for semantics and ILDSN for images, which are then fused by a Fully Connected Module (FCM). This joint discrimination enhances its ability to learn the joint probability distribution of image and semantic spaces.
  • Feature Extractor (Φ): A pre-trained VGG19 network is used as a feature extractor to compute perceptual loss, helping to eliminate facial ambiguity and recover missing details. It specifically uses low-level features from the 'Conv3_3' layer to capture complex edge textures.

A crucial aspect of SEGA-FURN is the use of Relativistic average Least Squares (RaLS) loss for adversarial training. This loss function, which combines the principles of Relativistic GANs (RaGAN) and Least Squares GANs (LSGAN), aims to alleviate gradient vanishing and improve training stability, leading to more authentic and visually pleasing SR images.

Datasets

The experiments were conducted on the CelebA dataset, a large-scale dataset containing 200K celebrity face images. For training, 162,048 HR face images were used, and the remaining 40,511 images served as the testing set. The images were resized and cropped to 256x256 pixels. LR images were generated using bicubic interpolation with downsampling factors of r=4 (producing 64x64 images) and r=8 (producing 32x32 images).

Key Insights and Ideas

  • Semantic Guidance is Key: The core innovation is using a semantic encoder to provide explicit semantic cues to the discriminator. This goes beyond just visual realism and helps the network understand facial structures, leading to more accurate and detailed reconstructions.
  • Joint Discrimination for Robustness: Discriminating on both image and semantic information simultaneously makes the discriminator more powerful and less susceptible to simple visual tricks, improving the overall FSR performance.
  • RaLS Loss for Stability: The adoption of RaLS loss is critical for stabilizing the GAN training process, a notorious challenge in super-resolution tasks. This leads to more consistent and reliable results.
  • Hierarchical RIDB Architecture: The Residual in Internal Dense Block (RIDB) architecture in the generator effectively leverages multi-level features, contributing to the generation of high-quality SR images.

Key Results

SEGA-FURN achieved state-of-the-art performance on the CelebA dataset for both 4x and 8x upscaling factors.

  • Quantitative:
    • 4x Upscaling: Achieved a PSNR of 30.14 dB and SSIM of 0.87.
  • 8x Upscaling: Achieved a PSNR of 25.21 dB and SSIM of 0.73. These results significantly outperformed existing methods, including SRGAN and ESRGAN, by a notable margin.

  • Qualitative: The method produced photo-realistic SR images that preserved perceptually sharper edges and fine facial textures, even for challenging 8x upscaling. Unlike other methods that suffered from over-smoothing, broken textures, or artifacts, SEGA-FURN generated visually appealing and faithful reconstructions. Ablation studies confirmed the significant contributions of the semantic encoder and the RaLS loss.

Limitations and Future Work

  • Limitations: While the paper demonstrates superior performance, the authors acknowledge that the method is primarily focused on face super-resolution. The generalizability to other image domains is not explicitly discussed. The computational cost of the VGG19 pre-trained networks for the semantic encoder and feature extractor might be a consideration for real-time applications on resource-constrained devices.
  • Future Work: The paper suggests that the proposed SEGA-FURN method can be extended to other domain-specific SR tasks. Further research could explore more efficient semantic encoder designs or investigate the application of this framework to video super-resolution.

Summary updated

Abstract

Face super-resolution is a domain-specific image super-resolution, which aims to generate High-Resolution (HR) face images from their Low-Resolution (LR) counterparts. In this paper, we propose a novel face super-resolution method, namely Semantic Encoder guided Generative Adversarial Face Ultra-Resolution Network (SEGA-FURN) to ultra-resolve an unaligned tiny LR face image to its HR counterpart with multiple ultra-upscaling factors (e.g., 4x and 8x). The proposed network is composed of a novel semantic encoder that has the ability to capture the embedded semantics to guide adversarial learning and a novel generator that uses a hierarchical architecture named Residual in Internal Dense Block (RIDB). Moreover, we propose a joint discriminator which discriminates both image data and embedded semantics. The joint discriminator learns the joint probability distribution of the image space and latent space. We also use a Relativistic average Least Squares loss (RaLS) as the adversarial loss to alleviate the gradient vanishing problem and enhance the stability of the training procedure. Extensive experiments on large face datasets have proved that the proposed method can achieve superior super-resolution results and significantly outperform other state-of-the-art methods in both qualitative and quantitative comparisons.
BibTeX
@misc{wang2022semantic,
  author = {Xiang Wang and Yimin Yang and Qixiang Pang and Xiao Lu and Yu Liu and Shan Du},
  title = {{Semantic Encoder Guided Generative Adversarial Face Ultra-Resolution Network}},
  year = {2022},
  eprint = {2211.10532},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2211.10532},
}
Figure