On the "steerability" of generative adversarial networks

Ali Jahanian, Lucy Chai, Phillip Isola

Abstract

While modern Generative Adversarial Networks (GANs) synthesize strikingly realistic images, they often fail to generalize beyond the biases of their training data. This paper investigates the "steerability" of GANs—their ability to undergo simple transformations like camera movements and color changes through latent space manipulation. We demonstrate that while GANs can be steered to perform these edits, the range of transformation is strictly bounded by the diversity of the training set. We introduce methods to quantify these limits and techniques to expand the steerable range.

Introduction: The Static Nature of GANs

In computer graphics, 3D engines provide total control over camera perspective and lighting. In contrast, deep generative models have traditionally been treated as "black boxes" that output static images. If a GAN generates a dog, can we move the camera closer or shift the dog to the side? The primary obstacle is dataset bias: if a model only sees centered dogs, it struggles to generate off-center ones. This research explores whether we can treat a GAN's latent space as a set of controls for a virtual camera.

Methodology: Navigating the Latent Space

To achieve steerability, we propose a self-supervised framework to identify specific directions, or "walks," in the GAN's latent space. We seek a vector $w$ such that moving from a latent code $z$ to $z + \alpha w$ corresponds to a predictable image transformation.

Our experiments compared simple linear walks (vector addition) against non-linear walks (parameterized by neural networks). Surprisingly, linear walks proved nearly as effective, suggesting that GANs naturally linearize many visual transformations during training. To measure success, we utilized external object detectors to track spatial shifts and color statistics to monitor luminance changes.

Empirical Findings: Limits and Priors

Our analysis reveals that a GAN's steerability is tethered to the "support" of its training data. For instance, a model can only zoom in on a specific object as much as the training data allows before the image quality collapses.

Semantic Priors and Hallucination

One of the most fascinating observations is how GANs use learned context to maintain realism during steering. When an object is shifted, the model must "fill in" the newly exposed areas.

Figure 1: Shifting a daisy vertically (Shift Y). As the flower moves, the GAN hallucinates grass at the bottom or sky at the top to maintain scene consistency.

As seen in Figure 1, shifting a daisy upward causes the GAN to generate more grass at the bottom, while shifting it down reveals more sky. This indicates that the model has learned a semantic prior about the environment in which daisies typically appear.

Color and Luminance Steering

Steerability also extends to non-spatial attributes like brightness. However, this is highly dependent on class-specific constraints. While a "jellyfish" can be steered through many colors, a "fire engine" is much harder to change because the model rarely sees it in colors other than red.

Figure 2: Steering the brightness of a generated building. The model successfully adjusts luminance while preserving structural details.

Key Results

  • Data Correlation: We found a strong correlation (up to 0.59) between the variance of an attribute in the training data and the model's steerability for that attribute.
  • Transformation Ranges: In BigGAN, we achieved horizontal/vertical shifts of $\pm 20\%$ and zooms up to 8x before significant quality degradation (measured by FID).
  • Enhanced Steerability: By jointly optimizing the generator weights with latent walks and using data augmentation, we significantly expanded the range of possible rotations and zooms compared to "frozen" models.

Conclusion and Future Work

While GANs can extrapolate slightly beyond their training distribution, they cannot create what they have never seen. The "steerability" of a model is a direct reflection of its training data's breadth. Future work should focus on designing generative architectures where steerability is an explicit objective, moving us closer to interactive, AI-driven graphics engines.

Summary updated

Abstract

An open secret in contemporary machine learning is that many models work beautifully on standard benchmarks but fail to generalize outside the lab. This has been attributed to biased training data, which provide poor coverage over real world events. Generative models are no exception, but recent advances in generative adversarial networks (GANs) suggest otherwise - these models can now synthesize strikingly realistic and diverse images. Is generative modeling of photos a solved problem? We show that although current GANs can fit standard datasets very well, they still fall short of being comprehensive models of the visual manifold. In particular, we study their ability to fit simple transformations such as camera movements and color changes. We find that the models reflect the biases of the datasets on which they are trained (e.g., centered objects), but that they also exhibit some capacity for generalization: by "steering" in latent space, we can shift the distribution while still creating realistic images. We hypothesize that the degree of distributional shift is related to the breadth of the training data distribution. Thus, we conduct experiments to quantify the limits of GAN transformations and introduce techniques to mitigate the problem. Code is released on our project page: https://ali-design.github.io/gan_steerability/
BibTeX
@inproceedings{jahanian2020steerability,
  author = {Ali Jahanian and Lucy Chai and Phillip Isola},
  title = {{On the "steerability" of generative adversarial networks}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=brQ6r9rZJo},
}
Figure