Rayleigh EigenDirections (REDs): GAN latent space traversals for multidimensional features

Guha Balakrishnan, Raghudeep Gadde, Aleix Martinez, Pietro PeronaarXiv 2022 · arXiv

Abstract

Navigating the latent spaces of Generative Adversarial Networks (GANs) typically involves moving along linear directions to edit scalar attributes. However, many semantic features, such as facial identity or specific objects, are multidimensional and reside on complex, curved manifolds. We introduce Rayleigh EigenDirections (REDs), a principled method to navigate these spaces by framing the search for traversal directions as a Rayleigh quotient maximization problem. This approach provides a closed-form solution that enables precise control over high-dimensional features while keeping others fixed, significantly outperforming standard linear methods.

Introduction

Generative models like StyleGAN2 organize semantic attributes within their latent spaces. Existing traversal techniques—methods used to move through this space to edit an image—often assume these attributes are simple scalars (e.g., "smile intensity") controllable via a single global linear direction.

In reality, many interesting image features are multidimensional. For instance, "facial identity" is often represented by a 512-dimensional embedding. These complex features do not follow straight lines in latent space; they reside on curved manifolds. Global linear models fail to capture this local complexity, leading to "attribute drift," where changing one feature (like hair) inadvertently alters another (like the person's face).

Methodology: Rayleigh EigenDirections

To address this, we seek a displacement $\delta z$ in the latent space that maximizes the change in a target feature $x$ while minimizing the change in a fixed feature $y$.

Linearization and the Rayleigh Quotient

We utilize Jacobians to approximate how small changes in the latent code $z$ affect features $x$ and $y$. The optimal direction is found by maximizing a Rayleigh quotient—a ratio of quadratic forms. This leads to a generalized eigenvalue problem: $A_c \delta z = \lambda A_f \delta z$, where $A_c$ and $A_f$ are matrices derived from the Jacobians of the changing and fixed features, respectively. The solutions to this equation are the Rayleigh EigenDirections (REDs).

Nonlinear Traversal via Projection

Because the latent space is curved, a single linear step eventually drifts from the desired manifold. We propose a "Projection" algorithm that recomputes the optimal REDs at every step, effectively "crawling" along the surface of the fixed-attribute manifold.

The Projection method recomputes REDs at each step to navigate the curved iso-identity manifold.

Experimental Results

We evaluated REDs using StyleGAN2 on the FFHQ face dataset and an in-house dataset of 100,000 high-resolution living room images.

Identity and Object Preservation

In the face domain, REDs allowed for changes in hairstyle or facial landmarks while keeping identity (measured by ArcFace) significantly more stable than baseline methods. In the living room domain, the method demonstrated impressive object-level control. We could keep a specific piece of furniture, such as a lamp or sofa, fixed while completely transforming the rest of the room's layout and style.

Object-preserving traversal: The lamp (red box) remains fixed while the room's style and layout change.

Frequency Control

REDs also successfully isolated changes to specific spatial frequency bands. This allows for modifications to "coarse" features like lighting and color while keeping "fine" features like textures and identity constant.

Discussion and Limitations

REDs provide a "black-box" friendly, closed-form solution using standard linear algebra, requiring no additional neural network training. However, some limitations remain:

  • Discrete Attributes: The method struggles with discrete attributes (e.g., "wearing eyeglasses") where local gradients may be near zero or noisy.
  • Latent Discontinuity: In complex scenes, the latent space can be discontinuous, leading to sudden "jumps" or artifacts if the traversal moves too far.
  • Computational Cost: Computing Jacobians via finite differences requires multiple forward passes of the generator, which may be slow for real-time applications without further optimization.

Future work may involve combining global directions for simple attributes with the local precision of REDs for complex, multidimensional features.

Summary updated

BibTeX
@misc{balakrishnan2022rayleigh,
  author = {Guha Balakrishnan and Raghudeep Gadde and Aleix Martinez and Pietro Perona},
  title = {{Rayleigh EigenDirections (REDs): GAN latent space traversals for multidimensional features}},
  year = {2022},
  eprint = {2201.10423},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2201.10423},
}
Figure