Low-Rank Subspaces in GANs

Jiapeng Zhu, Ruili Feng, Yujun Shen, Deli Zhao, Zheng-Jun Zha, Jingren Zhou, Qifeng Chen

TL;DR

LowRankGAN introduces a method for precise, local image editing in Generative Adversarial Networks (GANs) by performing low-rank factorization on the Jacobian matrix of the generator. By identifying the "null space" of specific image regions, researchers can manipulate local attributes (like eye shape or car color) without affecting the rest of the image—all without requiring explicit spatial masks during the editing process.

The Challenge of Local Editing in GANs

GANs learn to represent complex data in a "latent space." While we know this space contains directions that correspond to semantic changes (like adding a smile or changing hair color), these changes are usually global. If you try to change a subject's eyes, their nose and mouth might inadvertently shift as well.

Existing solutions for "local" editing typically rely on one of two restrictive approaches:

  1. Supervised Learning: Labeling thousands of images to find a specific boundary.
  2. Spatial Masks: Manually masking out parts of the image during the editing process to "force" locality.

The challenge is to discover these local semantic directions in an unsupervised manner that is robust, precise, and generalizes across different images without needing a pixel-perfect mask every time a change is made.

The LowRankGAN Methodology

The authors propose a framework called LowRankGAN to achieve this unsupervised local control. The technical core involves three main steps:

Overview of the LowRankGAN method, showing the Jacobian of the object, its low-rank subspace, and the null space used for local editing.
  1. The Jacobian Bridge: The method relates the latent space ($z$) to the pixel space ($x$) using the Jacobian matrix ($J_z$). This matrix describes how a tiny change in the latent code affects every single pixel in the output image.
  2. Low-Rank Factorization: Because GANs are often over-parameterized, the Jacobian is "degenerate" (low-rank). The authors use Principal Component Pursuit (PCP) to decompose the Jacobian into a low-rank matrix (representing the core semantic attributes) and a sparse matrix (representing noise).
  3. Null Space Projection: To edit a specific "Region A" (e.g., the car body) without touching "Region B" (the background), the system calculates the Jacobian for Region B. It then finds the null space of that Jacobian—the set of directions in the latent space that result in zero change to Region B. By projecting an attribute vector into this null space, they create a strictly "local" edit direction.

Key Insights

  • The Jacobian is Naturally Low-Rank: The authors provide a theoretical proof showing that the rank of the Jacobian is progressively reduced through the layers of a GAN. This confirms that attributes live on a low-dimensional manifold, making low-rank factorization the ideal mathematical tool.
  • One Sample is All You Need: Surprisingly, analyzing the Jacobian of a single synthesized image is enough to find a direction that works for the entire dataset. For example, a "car color" direction found using one car image successfully changes the color of completely different cars in other images.
  • Robustness to "Messy" Masks: Unlike previous methods that require precise segmentation, LowRankGAN works with rough bounding boxes. Because the latent space is semantically aware, the math "figures out" the underlying object even if the user's input mask is imprecise.
  • Precision Relaxation: Acknowledging that "perfect" local control can sometimes look unnatural (e.g., opening a mouth without moving the chin), the authors introduce a parameter ($r_{relax}$) to allow for a controlled amount of "bleed" into adjacent regions for more realistic results.

Results and Performance

The researchers validated LowRankGAN on state-of-the-art architectures like StyleGAN2 (FFHQ, LSUN Cars/Churches) and BigGAN (ImageNet).

  • Quantitative Superiority: LowRankGAN outperformed existing unsupervised methods like GANSpace and SeFa in terms of FID (image quality) and MSE (precision of local control). On StyleGAN2 faces, it achieved an FID of 8.01 compared to 9.14 for unprojected edits.
  • User Preference: In a user study involving 600 votes, LowRankGAN was preferred 56% of the time, significantly beating out SeFa (37.6%) and GANSpace (5.4%).
  • Data Augmentation: The method can generate "rare" samples (e.g., people with closed eyes, which are scarce in the original FFHQ dataset) to improve the training of downstream models.

Limitations and Future Work

While powerful, the method struggles with "pixel-level" control due to semantic entanglement. For example, it is difficult to edit only the left eye without the right eye, because the GAN's latent space tends to treat "eyes" as a single coupled semantic unit. Additionally, while the method localizes attributes well, users still need to manually inspect and identify which singular vector corresponds to which specific attribute (e.g., color vs. shape).

Future work may explore how this low-rank structure could be used to further compress GAN models or improve the efficiency of GAN inversion (mapping real images back into the latent space).

Summary updated

Abstract

The latent space of a Generative Adversarial Network (GAN) has been shown to encode rich semantics within some subspaces. To identify these subspaces, researchers typically analyze the statistical information from a collection of synthesized data, and the identified subspaces tend to control image attributes globally (i.e., manipulating an attribute causes the change of an entire image). By contrast, this work introduces low-rank subspaces that enable more precise control of GAN generation. Concretely, given an arbitrary image and a region of interest (e.g., eyes of face images), we manage to relate the latent space to the image region with the Jacobian matrix and then use low-rank factorization to discover steerable latent subspaces. There are three distinguishable strengths of our approach that can be aptly called LowRankGAN. First, compared to analytic algorithms in prior work, our low-rank factorization of Jacobians is able to find the low-dimensional representation of attribute manifold, making image editing more precise and controllable. Second, low-rank factorization naturally yields a null space of attributes such that moving the latent code within it only affects the outer region of interest. Therefore, local image editing can be simply achieved by projecting an attribute vector into the null space without relying on a spatial mask as existing methods do. Third, our method can robustly work with a local region from one image for analysis yet well generalize to other images, making it much easy to use in practice. Extensive experiments on state-of-the-art GAN models (including StyleGAN2 and BigGAN) trained on various datasets demonstrate the effectiveness of our LowRankGAN.
BibTeX
@inproceedings{zhu2021low,
  author = {Jiapeng Zhu and Ruili Feng and Yujun Shen and Deli Zhao and Zheng-Jun Zha and Jingren Zhou and Qifeng Chen},
  title = {{Low-Rank Subspaces in GANs}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=wALdFukHyYd},
}
Figure