Semantic-Aware Latent Space Exploration for Face Image Restoration

Yanhui Guo, Fangzhou Luo, Xiaolin Wu

Abstract: For image restoration, the majority of existing deep learning-based algorithms have a tendency to overfit the training data, resulting in poor performance when confronted with unseen degradations. To achieve more robust restoration, generative adversarial network (GAN) prior based methods have been proposed, demonstrating a promising capacity to restore photo-realistic and high-quality results. However, these methods are susceptible to semantic ambiguity, particularly with semantically relevant images such as facial images. In this paper, we propose a semantic-aware latent space exploration method for image restoration (SAIR). By explicitly modeling referenced semantics information, SAIR is able to reliably restore severely degraded images not only to high-resolution highly-realistic looks but also to correct semantics. Quantitative and qualitative experiments collectively demonstrate the effectiveness of the proposed SAIR.

Introduction

Face image restoration has evolved significantly with the advent of deep learning. However, traditional supervised methods often overfit to specific training degradations, failing when encountering real-world noise or blur. To address this, researchers have turned to Generative Adversarial Network (GAN) priors, such as StyleGAN, which provide a rich manifold of high-quality facial features. While these methods produce sharp images, they frequently suffer from "semantic confusion"—the restored face may look realistic but fails to match the original person's identity or expression. This is particularly problematic in severe degradation scenarios (e.g., 32x downsampling) where the input pixels provide insufficient guidance for the model to accurately reconstruct the target.

The SAIR Framework

The proposed Semantic-Aware latent space exploration for Image Restoration (SAIR) addresses this by explicitly modeling semantic information. SAIR operates by optimizing a latent code $w$ within the $W+$ space of a pre-trained StyleGAN, guided by three primary semantic anchors:

  1. Identity Semantics: By utilizing a single high-quality reference image of the target individual from a candidate pool, SAIR constrains the optimization process. It minimizes the distance between the restored output and the reference in a deep feature space (using ArcFace), ensuring the restored face retains the correct identity.
  2. Emotion Semantics: Facial expressions are often lost in heavily degraded inputs. SAIR utilizes pre-defined emotion hyperplanes in the latent space to push the optimization toward the correct expression cluster (e.g., happiness, sadness) inferred from the low-quality input.
  3. Skin Color Semantics: To prevent the model from defaulting to an "average" skin tone, a differentiable histogram loss is employed to maintain consistency with the original low-quality observation.
The SAIR framework overview, illustrating the integration of reference images and latent space optimization to achieve semantically consistent face restoration.

Experimental Results

The effectiveness of SAIR was evaluated on a dataset of 400 high-quality celebrity images subjected to various degradations. Quantitatively, SAIR achieved a Fréchet Inception Distance (FID) of 108.7, outperforming competitors like PULSE (143.1) and GPEN (171.9). More importantly, SAIR's identity similarity score of 0.54 was significantly higher than GPEN (0.27) and PULSE (0.16), confirming its ability to preserve recognizable features.

Qualitatively, SAIR produces results that are not only sharp but also faithful to the original subject's expression and identity. As shown in comparative studies, SAIR avoids the common pitfalls of other generative methods that often "hallucinate" entirely different people or neutral expressions when the original subject was clearly displaying a specific emotion.

Qualitative comparison of SAIR against state-of-the-art methods. SAIR demonstrates superior preservation of identity and facial expression compared to existing generative prior-based approaches.

Conclusion

SAIR represents a significant step forward in making GAN-based face restoration practical for real-world applications where identity preservation is paramount. By "squeezing" the solution space using identity and emotion anchors, the model moves beyond mere pixel reconstruction toward semantic fidelity. While the iterative optimization process is currently slower than feed-forward alternatives (taking approximately 100 seconds per image on a TITAN Xp GPU), the gain in semantic accuracy is substantial. Future work will focus on accelerating the optimization and extending the semantic-aware framework to other object categories like cats or cars.

Summary updated

Abstract

For image restoration, the majority of existing deep learning-based algorithms have a tendency to overfit the training data, resulting in poor performance when confronted with unseen degradations. To achieve more robust restoration, generative adversarial network (GAN) prior based methods have been proposed, demonstrating a promising capacity to restore photo-realistic and high-quality results. However, these methods are susceptible to semantic ambiguity, particularly with semantically relevant images such as facial images. In this paper, we propose a semantic-aware latent space exploration method for image restoration (SAIR). By explicitly modeling referenced semantics information, SAIR is able to reliably restore severely degraded images not only to high-resolution highly-realistic looks but also to correct semantics. Quantitative and qualitative experiments collectively demonstrate the effectiveness of the proposed SAIR. Our code can be found in https://github.com/Liamkuo/SAIR.
BibTeX
@inproceedings{guo2022semantic,
  author = {Yanhui Guo and Fangzhou Luo and Xiaolin Wu},
  title = {{Semantic-Aware Latent Space Exploration for Face Image Restoration}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  doi = {10.48550/arxiv.2203.03005},
}

Loading related content...

Figure