Face Super-Resolution Through Wasserstein GANs

Zhimin Chen, Yuguang TongarXiv 2017 · arXiv

Abstract

Generative Adversarial Networks (GANs) are powerful tools for image synthesis but suffer from training instability and uninformative loss metrics. This paper evaluates the application of Wasserstein GANs (WGAN) and WGAN with Gradient Penalty (WGAN-GP) to the task of face super-resolution. By reconstructing 64x64 high-resolution faces from 16x16 inputs using the CelebA dataset, we demonstrate that WGAN-GP provides superior training stability across architectures and a loss metric that correlates with perceptual quality, effectively mitigating common issues like mode collapse.

Introduction

Single Image Super-Resolution (SISR) aims to recover high-frequency details from low-resolution inputs. While standard GANs can produce sharp images, they are notoriously difficult to train due to three primary hurdles: training instability, uninformative loss values, and mode collapse. This study investigates whether Wasserstein-based objectives can overcome these challenges in the specific domain of facial reconstruction.

Methodology

The researchers compared three primary training objectives:

  1. Standard GAN: Utilizing Jensen-Shannon (JS) divergence, which often leads to vanishing gradients or oscillating training.
  2. WGAN (Weight Clipping): Approximating Wasserstein distance by enforcing a 1-Lipschitz constraint via weight clipping.
  3. WGAN-GP (Gradient Penalty): An improved WGAN variant that enforces the Lipschitz constraint by penalizing the gradient norm of the discriminator (critic).

To ensure the reconstructed faces remained faithful to the low-resolution inputs, an $L_1$ perceptual loss term was added to the objective. This term penalizes the pixel-wise difference between the downsampled generated image and the original input. Experiments were conducted on the CelebA dataset, downsampling 64x64 labels to 16x16 inputs.

Training Stability and Loss Interpretability

A core finding of this work is the reliability of the Wasserstein loss metric. In traditional GANs, the discriminator loss rarely reflects the actual quality of generated images. In contrast, the loss in WGAN-GP decreases monotonically as the model improves, providing a clear signal for convergence and hyperparameter tuning.

Comparison of L1 norm, Discriminator loss, and Generator loss across different GAN variants.

Furthermore, WGAN-GP demonstrated remarkable robustness to architectural changes. While standard GANs failed when using simple Multi-Layer Perceptrons (MLPs) or removing Batch Normalization, WGAN-GP consistently produced reasonable results, suggesting it significantly lowers the "black magic" required for GAN architecture design.

Experimental Results

Qualitatively, both WGAN and WGAN-GP outperformed traditional bicubic interpolation and standard GANs. The Wasserstein-based models produced sharper features, such as more defined eyes and realistic accessories like glasses, which were often blurred or distorted in standard GAN outputs.

Qualitative comparison of face super-resolution results across different methods.

Quantitatively, WGAN-GP effectively addressed mode collapse. When generating images from pure noise, WGAN-GP produced a diverse array of facial features and expressions, whereas standard GANs tended to collapse toward a limited set of "safe" variations.

Discussion and Future Work

Despite the improvements, several challenges remain. The models showed a bias toward frontal faces, likely due to the distribution of the CelebA dataset, performing less effectively on side profiles. Additionally, while the $L_1$ loss ensures input fidelity, it sometimes results in noisier images compared to purely adversarial objectives, highlighting a trade-off between pixel-wise accuracy and human visual preference. Future work could explore more diverse datasets and hybrid loss functions to further refine the balance between sharpness and structural integrity.

Summary updated

Abstract

Generative adversarial networks (GANs) have received a tremendous amount of attention in the past few years, and have inspired applications addressing a wide range of problems. Despite its great potential, GANs are difficult to train. Recently, a series of papers (Arjovsky & Bottou, 2017a; Arjovsky et al. 2017b; and Gulrajani et al. 2017) proposed using Wasserstein distance as the training objective and promised easy, stable GAN training across architectures with minimal hyperparameter tuning. In this paper, we compare the performance of Wasserstein distance with other training objectives on a variety of GAN architectures in the context of single image super-resolution. Our results agree that Wasserstein GAN with gradient penalty (WGAN-GP) provides stable and converging GAN training and that Wasserstein distance is an effective metric to gauge training progress.
BibTeX
@misc{chen2017face,
  author = {Zhimin Chen and Yuguang Tong},
  title = {{Face Super-Resolution Through Wasserstein GANs}},
  year = {2017},
  eprint = {1705.02438},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1705.02438},
}
Figure