Improved StyleGAN-v2 based Inversion for Out-of-Distribution Images

Rakshith Subramanyam, Vivek Sivaraman Narayanaswamy, Mark Naufel, Andreas Spanias, Jayaraman J. ThiagarajanICML 2022 · OpenReview

Generative Adversarial Networks (GANs), particularly StyleGAN-v2, are incredibly adept at synthesizing realistic images. To use these models for editing or restoring real-world photos, researchers rely on GAN inversion—the process of finding the specific "latent code" that, when fed into the generator, reproduces a target image.

While current inversion methods work well for images that closely resemble the original training data (e.g., perfectly aligned faces), they struggle significantly with "out-of-distribution" (OOD) data. If a face is rotated, scaled, or comes from an entirely different domain like a cartoon or a medical scan, standard inversion techniques often produce blurry, artifact-heavy results. This failure stems from the highly non-robust nature of the latent spaces typically used for optimization.

To solve this, researchers have introduced SPHInX (StyleGAN with Projection Heads for Inverting X), a new framework that significantly improves GAN inversion for OOD images.

How SPHInX Works

Instead of directly searching for a single latent vector, SPHInX optimizes projection heads—small neural networks that map from a simple prior distribution to the generator's latent space.

Overview of the SPHInX architecture.

The core components of the SPHInX architecture include:

  1. Style Projection Head ($P_s$): This replaces the standard StyleGAN mapping function. It utilizes a bottleneck architecture with independent decoders for each of the generator's 18 layers. This "decoupling" gives the model the flexibility needed to handle OOD features that don't follow the rules of the original training distribution.
  2. Content Projection Head ($P_c$): Instead of using a fixed constant tensor for the generator's starting point, SPHInX learns a content projection to better capture the unique semantic structures of OOD images.
  3. Vicinal Regularization: During optimization, the model samples multiple realizations from a prior distribution in every iteration. This forces the projection head to find a robust "neighborhood" (vicinity) of valid solutions, rather than a single fragile point that might easily break under perturbation.

Key Insights and Versatility

The researchers evaluated SPHInX across a diverse range of datasets, including in-the-wild faces with geometric shifts, animal faces, and medical imaging (like chest X-rays and retinal scans).

One of the most striking findings is the versatility of the "Face Prior." The study demonstrates that a GAN trained exclusively on human faces can serve as a powerful prior for entirely different domains, such as medical images, provided the inversion process is robust enough. This suggests that the low-level features learned by StyleGAN are far more universal than previously thought.

Performance and Results

SPHInX consistently outperforms state-of-the-art baselines in both qualitative and quantitative measures, such as PSNR and LPIPS.

  • Geometric Shifts: In tests involving rotated or scaled faces, SPHInX maintains high fidelity where other methods produce significant artifacts or fail to capture the shift entirely.
  • Medical Applications: SPHInX achieves high-quality reconstructions in medical imaging tasks like denoising and compressed sensing. For instance, it produced high-fidelity chest X-ray reconstructions with an average PSNR of ~34 dB, even when using only 1% of the original measurements.
  • Semantic Editing: Despite handling OOD images, SPHInX still allows for semantic manipulations (e.g., adding a smile to a cartoon) using existing StyleGAN attribute directions without requiring specialized encoders.
Comparison of reconstruction performance on a rotated face.

Limitations

The primary limitation of SPHInX is its computational cost. Because it optimizes projection heads per image for thousands of iterations, it is significantly slower than "encoder-based" methods that offer near-instantaneous inversion. Additionally, while the results on medical images are impressive, using a "face-trained" prior for clinical diagnosis carries inherent risks that require further investigation into potential cross-domain biases.

Summary updated

Abstract

Inverting an image onto the latent space of pre-trained generators, e.g., StyleGAN-v2, has emerged as a popular strategy to leverage strong image priors for ill-posed restoration. Several studies h...
BibTeX
@inproceedings{subramanyam2022improved,
  author = {Rakshith Subramanyam and Vivek Sivaraman Narayanaswamy and Mark Naufel and Andreas Spanias and Jayaraman J. Thiagarajan},
  title = {{Improved StyleGAN-v2 based Inversion for Out-of-Distribution Images}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=NY0i8ZXkOr},
}
Figure