Style-Based Unsupervised Learning for Real-World Face Image Super-Resolution

Ahmed Cheikh Sidiya, Xin LiIntechOpen 2020 · DOI · OpenAlex

Abstract

Face image synthesis has advanced rapidly in recent years. However, similar success has not been witnessed in related areas such as face single image super-resolution (SISR). The performance of SISR on real-world low-quality face images remains unsatisfactory. In this paper, we demonstrate how to advance the state-of-the-art in face SISR by leveraging style-based generator in unsupervised settings. For real-world low-resolution (LR) face images, we propose a novel unsupervised learning approach by combining style-based generator with relativistic discriminator. With a carefully designed training strategy, we demonstrate our converges faster and better suppresses artifacts than Bulat’s approach. When trained on an ensemble of high-quality datasets (CelebA, AFLW, LS3D-W, and VGGFace2), we report significant visual quality improvements over other competing methods especially for real-world low-quality face images such as those in Widerface. Additionally, we have verified that both our unsupervised approaches are capable of improving the matching performance of widely used face recognition systems such as OpenFace.

Introduction: The Challenge of Real-World Super-Resolution

Most Single Image Super-Resolution (SISR) models are traditionally trained using supervised learning, which requires perfectly aligned pairs of high-resolution (HR) and low-resolution (LR) images. Because real-world pairs are difficult to capture, researchers often rely on synthetic LR images created via mathematical downsampling (e.g., bicubic interpolation).

However, real-world degradation—comprising sensor noise, motion blur, and compression artifacts—is far more complex than simple mathematical functions. Models trained on "clean" artificial data frequently fail when applied to "messy" real-world photos. This paper addresses Blind SISR, a scenario where the degradation model is unknown and paired training data is unavailable.

The traditional SISR pipeline involves patch extraction, non-linear mapping, and reconstruction to transform low-resolution inputs into high-resolution outputs.

Methods: An Unsupervised Style-Transfer Approach

The authors propose an unsupervised framework based on the CycleGAN architecture, enhanced with several key innovations to bridge the "domain gap" between synthetic and real-world images:

  1. Asymmetrical CycleGAN: The system uses two generative adversarial networks (GANs). The High-to-Low (H2L) network learns to "degrade" HR images into the style of real-world LR images, effectively modeling realistic degradation. The Low-to-High (L2H) network then performs the super-resolution task.
  2. Style-Based Generator: Inspired by StyleGAN, this architecture is better suited for capturing the intricate nuances of facial features compared to standard CNNs.
  3. Relativistic Discriminator: Instead of judging images in isolation, this discriminator estimates the probability that a real image is more realistic than a fake one, leading to more stable training and fewer artifacts.
  4. Self-Attention Layers: These layers ensure global structural consistency, preventing common issues like mismatched facial features (e.g., asymmetrical eyes).
  5. Ablation-Driven Strategy: The authors found that standard techniques like Batch Normalization and data augmentation (flipping/rotating) actually introduced artifacts in this unsupervised setting and were intentionally omitted.

Datasets and Training

To ensure a robust representation of human faces, the model was trained on a massive ensemble of datasets:

  • High-Resolution (HR) Training: Over 229,000 images from CelebA, AFLW, LS3D-W, and VGGFace2.
  • Real Low-Resolution (RLR) Training: Over 156,000 images from Widerface, a dataset characterized by extreme variability in scale, pose, and occlusion.

Key Insights: Degradation as "Style"

The core innovation of this work is treating image degradation as a "style" to be learned rather than a fixed mathematical function. By training a generator to transform clean faces into "Widerface-style" faces, the model learns the specific noise and blur patterns inherent in real-world camera sensors. Furthermore, the study highlights a "Normalization Paradox," where standard deep learning staples like Batch Normalization can be counterproductive in unsupervised SISR, often hallucinating artifacts.

Results and Evaluation

The proposed method demonstrates significant qualitative and quantitative improvements over existing state-of-the-art approaches:

  • Qualitative Superiority: On the Widerface dataset, the model outperformed supervised models (like FSRGAN) and previous unsupervised methods (Bulat’s method), successfully avoiding artifacts like "gender swapping" or "age variation."
  • Quantitative Recognition: Using OpenFace to test machine recognizability, the unsupervised approach achieved a 94.85% True Positive Rate on compressed data, significantly higher than the supervised version's 91.64%.
  • Metric Performance: On artificial LR data, the supervised version of the proposed architecture reached 23.65 dB PSNR, outperforming FSRGAN (22.84 dB).
Comparison of different super-resolution methods. While metrics like PSNR provide a baseline, this paper emphasizes that visual quality and machine recognizability are more critical for real-world applications.

Limitations and Future Work

Despite its success, the model remains sensitive to extreme face poses and severe occlusions. The High-to-Low network currently lacks sufficient training data for "edge cases," such as faces turned 90 degrees or those heavily obscured by hands. Future research will focus on expanding training diversity and potentially incorporating 3D facial priors to better manage extreme geometry.

Summary updated

Abstract

Face image synthesis has advanced rapidly in recent years. However, similar success has not been witnessed in related areas such as face single image super-resolution (SISR). The performance of SISR on real-world low-quality face images remains unsatisfactory. In this paper, we demonstrate how to advance the state-of-the-art in face SISR by leveraging style-based generator in unsupervised settings. For real-world low-resolution (LR) face images, we propose a novel unsupervised learning approach by combining style-based generator with relativistic discriminator. With a carefully designed training strategy, we demonstrate our converges faster and better suppresses artifacts than Bulat’s approach. When trained on an ensemble of high-quality datasets (CelebA, AFLW, LS3D-W, and VGGFace2), we report significant visual quality improvements over other competing methods especially for real-world low-quality face images such as those in Widerface. Additionally, we have verified that both our unsupervised approaches are capable of improving the matching performance of widely used face recognition systems such as OpenFace.
BibTeX
@inproceedings{sidiya2020style,
  author = {Ahmed Cheikh Sidiya and Xin Li},
  title = {{Style-Based Unsupervised Learning for Real-World Face Image Super-Resolution}},
  booktitle = {IntechOpen},
  year = {2020},
  doi = {10.5772/intechopen.92320},
}
Figure