Low-Rank Subspaces in GANs

Jiapeng Zhu, Ruili Feng, Yujun Shen, Deli Zhao, Zhengjun Zha, Jingren Zhou, Qifeng Chen

Abstract

The latent space of a Generative Adversarial Network (GAN) has been shown to encode rich semantics within some subspaces. To identify these subspaces, researchers typically analyze the statistical information from a collection of synthesized data, and the identified subspaces tend to control image attributes globally. By contrast, this work introduces low-rank subspaces that enable more precise control of GAN generation. Concretely, given an arbitrary image and a region of interest (ROI), we relate the latent space to the image region with the Jacobian matrix and then use low-rank factorization to discover steerable latent subspaces. Our approach, LowRankGAN, offers three key strengths: precise attribute manifold representation, mask-free local editing via null space projection, and robust generalization from single-sample analysis.

Introduction

Generative Adversarial Networks (GANs) have achieved remarkable success in synthesizing high-fidelity images. A significant discovery in GAN research is that their latent spaces—the mathematical domains of input codes—are semantically organized. However, most existing methods for identifying "editing directions" in these spaces are global in nature. For instance, moving along a direction identified for "smiling" might inadvertently alter a subject's hair, skin tone, or background.

Achieving local control (e.g., editing only the eyes or a car's color) has traditionally required spatial masks or heavy supervision with labeled data. LowRankGAN addresses this challenge by finding precise, local editing directions in an unsupervised manner, using only the latent code and without requiring pixel-perfect masks at inference time.

Methodology

The LowRankGAN framework consists of three primary technical components:

  1. Jacobian Mapping: For a given image and a specified region of interest (ROI), the method computes the Jacobian matrix. This matrix mathematically defines how infinitesimal changes in the latent code $z$ translate to changes in the pixel values of the ROI.
  2. Low-Rank Factorization: Recognizing that the raw Jacobian is often noisy and redundant, the authors apply Robust Principal Component Analysis (RPCA). This decomposes the Jacobian into a low-rank matrix—representing the core semantic attributes—and a sparse matrix representing noise. This process uncovers the intrinsic, low-dimensional manifold of the attributes.
  3. Null Space Projection: This is the core innovation for local editing. By identifying the "null space" of regions outside the ROI, an attribute vector (e.g., "eye-closing") can be projected into a subspace where it has minimal to zero effect on the background or non-target regions. This enables "mask-free" local editing.
Overview of the LowRankGAN framework: (a) Jacobian mapping of an object, (b) identifying the low-rank subspace, and (c) utilizing the null space for isolated editing.

The method also incorporates precision relaxation ($r_{relax}$), a parameter that allows users to balance perfect isolation with the "naturalness" of an edit, acknowledging that some physical features (like jaw movement during speaking) are naturally entangled.

Experimental Evaluation

The researchers evaluated LowRankGAN on several high-resolution datasets, including FFHQ (faces), LSUN (Church and Car), ImageNet (diverse objects), and MetFaces (artistic portraits).

Key Results: * Quantitative Performance: LowRankGAN outperformed unsupervised baselines like GANSpace and SeFa in terms of Fréchet Inception Distance (FID) and Mean Squared Error (MSE) for local changes. A user study indicated a 56% preference for LowRankGAN over competitors. * Qualitative Precision: The method demonstrated the ability to modify specific attributes—such as a bird's pose or a car's color—while maintaining a perfectly static background. * Data Augmentation: By generating 29,800 synthetic images with specific local traits (e.g., closed eyes), the authors improved the accuracy of a specialized classifier from 52% to 95%, demonstrating the utility of precise editing for balancing datasets.

Discussion and Insights

LowRankGAN reveals several important properties of GAN latent spaces:

  • The Utility of the Null Space: While prior work focused on principal variations, the null space—directions that do not change specific pixels—is essential for precise local control.
  • Single-Sample Analysis: Unlike statistical methods requiring thousands of images, LowRankGAN can identify effective editing directions by analyzing the Jacobian of just one representative image.
  • Semantic Generalization: Directions found for a specific object (e.g., a car body) generalize well to other images in the same category, suggesting a consistent intrinsic semantic structure within the GAN.

Conclusion and Limitations

While LowRankGAN provides unprecedented local control, it operates on semantic regions rather than individual pixels. It may struggle with extremely fine-grained tasks, such as closing only one eye. Additionally, the method's effectiveness is constrained by the pre-trained GAN's learned representations; it cannot discover attributes the model has not captured. Future work may explore deeper integration with physical constraints to better handle naturally entangled attributes.

Summary updated

Abstract

The latent space of a Generative Adversarial Network (GAN) has been shown to encode rich semantics within some subspaces. To identify these subspaces, researchers typically analyze the statistical information from a collection of synthesized data, and the identified subspaces tend to control image attributes globally (i.e., manipulating an attribute causes the change of an entire image). By contrast, this work introduces low-rank subspaces that enable more precise control of GAN generation. Concretely, given an arbitrary image and a region of interest (e.g., eyes of face images), we manage to relate the latent space to the image region with the Jacobian matrix and then use low-rank factorization to discover steerable latent subspaces. There are three distinguishable strengths of our approach that can be aptly called LowRankGAN. First, compared to analytic algorithms in prior work, our low-rank factorization of Jacobians is able to find the low-dimensional representation of attribute manifold, making image editing more precise and controllable. Second, low-rank factorization naturally yields a null space of attributes such that moving the latent code within it only affects the outer region of interest. Therefore, local image editing can be simply achieved by projecting an attribute vector into the null space without relying on a spatial mask as existing methods do. Third, our method can robustly work with a local region from one image for analysis yet well generalize to other images, making it much easy to use in practice. Extensive experiments on state-of-the-art GAN models (including StyleGAN2 and BigGAN) trained on various datasets demonstrate the effectiveness of our LowRankGAN.
BibTeX
@inproceedings{zhu2021low,
  author = {Jiapeng Zhu and Ruili Feng and Yujun Shen and Deli Zhao and Zhengjun Zha and Jingren Zhou and Qifeng Chen},
  title = {{Low-Rank Subspaces in GANs}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=wALdFukHyYd},
}
Figure