Abstract
While modern Generative Adversarial Networks (GANs) synthesize strikingly realistic images, they often fail to generalize beyond the biases of their training data. This paper investigates the "steerability" of GANs—their ability to undergo simple transformations like camera movements and color changes through latent space manipulation. We demonstrate that while GANs can be steered to perform these edits, the range of transformation is strictly bounded by the diversity of the training set. We introduce methods to quantify these limits and techniques to expand the steerable range.
Introduction: The Static Nature of GANs
In computer graphics, 3D engines provide total control over camera perspective and lighting. In contrast, deep generative models have traditionally been treated as "black boxes" that output static images. If a GAN generates a dog, can we move the camera closer or shift the dog to the side? The primary obstacle is dataset bias: if a model only sees centered dogs, it struggles to generate off-center ones. This research explores whether we can treat a GAN's latent space as a set of controls for a virtual camera.
Methodology: Navigating the Latent Space
To achieve steerability, we propose a self-supervised framework to identify specific directions, or "walks," in the GAN's latent space. We seek a vector $w$ such that moving from a latent code $z$ to $z + \alpha w$ corresponds to a predictable image transformation.
Our experiments compared simple linear walks (vector addition) against non-linear walks (parameterized by neural networks). Surprisingly, linear walks proved nearly as effective, suggesting that GANs naturally linearize many visual transformations during training. To measure success, we utilized external object detectors to track spatial shifts and color statistics to monitor luminance changes.
Empirical Findings: Limits and Priors
Our analysis reveals that a GAN's steerability is tethered to the "support" of its training data. For instance, a model can only zoom in on a specific object as much as the training data allows before the image quality collapses.
Semantic Priors and Hallucination
One of the most fascinating observations is how GANs use learned context to maintain realism during steering. When an object is shifted, the model must "fill in" the newly exposed areas.
As seen in Figure 1, shifting a daisy upward causes the GAN to generate more grass at the bottom, while shifting it down reveals more sky. This indicates that the model has learned a semantic prior about the environment in which daisies typically appear.
Color and Luminance Steering
Steerability also extends to non-spatial attributes like brightness. However, this is highly dependent on class-specific constraints. While a "jellyfish" can be steered through many colors, a "fire engine" is much harder to change because the model rarely sees it in colors other than red.
Key Results
- Data Correlation: We found a strong correlation (up to 0.59) between the variance of an attribute in the training data and the model's steerability for that attribute.
- Transformation Ranges: In BigGAN, we achieved horizontal/vertical shifts of $\pm 20\%$ and zooms up to 8x before significant quality degradation (measured by FID).
- Enhanced Steerability: By jointly optimizing the generator weights with latent walks and using data augmentation, we significantly expanded the range of possible rotations and zooms compared to "frozen" models.
Conclusion and Future Work
While GANs can extrapolate slightly beyond their training distribution, they cannot create what they have never seen. The "steerability" of a model is a direct reflection of its training data's breadth. Future work should focus on designing generative architectures where steerability is an explicit objective, moving us closer to interactive, AI-driven graphics engines.
Summary updated