TL;DR
This paper introduces SEGA-FURN, a novel face super-resolution network that leverages a semantic encoder to guide the generative adversarial process. By incorporating semantic information alongside visual data, SEGA-FURN achieves state-of-the-art results in generating high-resolution face images from extremely low-resolution inputs, even with unaligned faces.
What Problem Does This Paper Address?
Face Super-Resolution (FSR) is a challenging task in computer vision, aiming to reconstruct high-resolution (HR) face images from low-resolution (LR) counterparts. Existing methods often struggle with complex facial expressions, lost facial information in LR images, and the need for precise facial alignment or prior knowledge like landmarks. Many deep learning-based FSR methods, while improving results, can produce overly smooth images due to reconstruction losses (L1/L2). Generative Adversarial Networks (GANs) offer a path to more perceptually realistic results, but they are prone to training instability, gradient vanishing, and model collapse. Furthermore, prior knowledge-based methods, while effective, demand significant computational resources and are not suitable for real-time applications. This paper addresses these limitations by proposing a GAN-based FSR method that is robust to unaligned faces, avoids explicit prior knowledge, and enhances training stability for superior FSR.
Methods
The proposed SEGA-FURN is a GAN-based framework with four key components:
- Semantic Encoder (E): This novel component extracts "embedded semantics" from both HR and LR face images. Unlike previous methods that solely rely on visual information, this encoder captures facial attributes (shape, gender) and spatial relationships, which are then used to guide the discriminator. It utilizes pre-trained VGG19 networks, adapted for different input dimensions.
- Generator (G): The generator employs a hierarchical architecture called Residual in Internal Dense Block (RIDB). This design, building upon previous work, effectively extracts and fuses hierarchical features, enhancing the generator's capability. It consists of a Shallow Feature Module (SFM), a Multi-level Dense Block Module (MDBM) composed of DNBs (which contain RIDBs), and an Upsampling Module (UM).
- Joint Discriminator (D): This discriminator is designed to evaluate tuples containing both visual information (HR/SR images) and embedded semantics. It comprises two sub-nets: ESLDSN for semantics and ILDSN for images, which are then fused by a Fully Connected Module (FCM). This joint discrimination enhances its ability to learn the joint probability distribution of image and semantic spaces.
- Feature Extractor (Φ): A pre-trained VGG19 network is used as a feature extractor to compute perceptual loss, helping to eliminate facial ambiguity and recover missing details. It specifically uses low-level features from the 'Conv3_3' layer to capture complex edge textures.
A crucial aspect of SEGA-FURN is the use of Relativistic average Least Squares (RaLS) loss for adversarial training. This loss function, which combines the principles of Relativistic GANs (RaGAN) and Least Squares GANs (LSGAN), aims to alleviate gradient vanishing and improve training stability, leading to more authentic and visually pleasing SR images.
Datasets
The experiments were conducted on the CelebA dataset, a large-scale dataset containing 200K celebrity face images. For training, 162,048 HR face images were used, and the remaining 40,511 images served as the testing set. The images were resized and cropped to 256x256 pixels. LR images were generated using bicubic interpolation with downsampling factors of r=4 (producing 64x64 images) and r=8 (producing 32x32 images).
Key Insights and Ideas
- Semantic Guidance is Key: The core innovation is using a semantic encoder to provide explicit semantic cues to the discriminator. This goes beyond just visual realism and helps the network understand facial structures, leading to more accurate and detailed reconstructions.
- Joint Discrimination for Robustness: Discriminating on both image and semantic information simultaneously makes the discriminator more powerful and less susceptible to simple visual tricks, improving the overall FSR performance.
- RaLS Loss for Stability: The adoption of RaLS loss is critical for stabilizing the GAN training process, a notorious challenge in super-resolution tasks. This leads to more consistent and reliable results.
- Hierarchical RIDB Architecture: The Residual in Internal Dense Block (RIDB) architecture in the generator effectively leverages multi-level features, contributing to the generation of high-quality SR images.
Key Results
SEGA-FURN achieved state-of-the-art performance on the CelebA dataset for both 4x and 8x upscaling factors.
- Quantitative:
- 4x Upscaling: Achieved a PSNR of 30.14 dB and SSIM of 0.87.
-
8x Upscaling: Achieved a PSNR of 25.21 dB and SSIM of 0.73. These results significantly outperformed existing methods, including SRGAN and ESRGAN, by a notable margin.
-
Qualitative: The method produced photo-realistic SR images that preserved perceptually sharper edges and fine facial textures, even for challenging 8x upscaling. Unlike other methods that suffered from over-smoothing, broken textures, or artifacts, SEGA-FURN generated visually appealing and faithful reconstructions. Ablation studies confirmed the significant contributions of the semantic encoder and the RaLS loss.
Limitations and Future Work
- Limitations: While the paper demonstrates superior performance, the authors acknowledge that the method is primarily focused on face super-resolution. The generalizability to other image domains is not explicitly discussed. The computational cost of the VGG19 pre-trained networks for the semantic encoder and feature extractor might be a consideration for real-time applications on resource-constrained devices.
- Future Work: The paper suggests that the proposed SEGA-FURN method can be extended to other domain-specific SR tasks. Further research could explore more efficient semantic encoder designs or investigate the application of this framework to video super-resolution.
Summary updated