Abstract
Generative Adversarial Networks (GANs) are able to generate high-quality images, but it remains difficult to explicitly specify the semantics of synthesized images. In this work, we aim to better understand the semantic representation of GANs, and thereby enable semantic control in GAN's generation process. Interestingly, we find that a well-trained GAN encodes image semantics in its internal feature maps in a surprisingly simple way: a linear transformation of feature maps suffices to extract the generated image semantics. To verify this simplicity, we conduct extensive experiments on various GANs and datasets; and thanks to this simplicity, we are able to learn a semantic segmentation model for a trained GAN from a small number (e.g., 8) of labeled images. Last but not least, leveraging our finding, we propose two few-shot image editing approaches, namely Semantic-Conditional Sampling and Semantic Image Editing. Given a trained GAN and as few as eight semantic annotations, the user is able to generate diverse images subject to a user-provided semantic layout, and control the synthesized image semantics.
The Challenge of Semantic Control in GANs
Generative Adversarial Networks (GANs) have achieved remarkable success in synthesizing photo-realistic images. However, "steering" these models remains a significant challenge. If a user wishes to generate a face with a specific nose shape or a bedroom with a particular furniture layout, they typically must train a complex Conditional GAN (cGAN) from scratch using massive, labeled datasets.
The core mystery addressed in this research is: How does a well-trained GAN internally represent the layout and parts of an image? While previous studies identified individual neurons responsible for specific concepts (like "trees"), these findings were often low-resolution and lacked a unified framework for extracting full semantic maps. This paper investigates whether a simple, universal mathematical relationship exists between a GAN's internal feature maps and the final image's semantic structure.
Methods: The Linear Semantic Extractor (LSE)
The authors introduce the Linear Semantic Extractor (LSE), an elegantly simple approach to decoding GAN representations. The process involves three primary steps:
- Feature Stacking: Internal feature maps are extracted from all layers of a trained GAN (such as StyleGAN2).
- Upsampling: Because different layers operate at different resolutions, all maps are upsampled to match the final image resolution.
- Linear Transformation: These maps are stacked into a single high-dimensional tensor. The researchers demonstrate that a simple linear transformation—a single matrix multiplication—is sufficient to classify every pixel into a semantic category (e.g., "eye," "grass," "wall").
Because this relationship is linear, it requires very little data to learn. The authors demonstrate few-shot learning, where the LSE is trained on a tiny handful of images (as few as 8 to 16) annotated by a pre-trained segmentation model like UNet or DeepLabV3. This LSE then serves as a bridge for applications like Semantic Image Editing (SIE), where a user modifies a mask to update the image, and Semantic Conditional Sampling (SCS), which generates new images from a provided layout.
Key Insights and Ideas
- The Linearity Discovery: The most striking finding is that non-linear extractors (multi-layer neural networks) perform no significantly better than the simple linear LSE. This suggests that GANs naturally organize their internal representations linearly.
- Hyper-cone Geometry: The authors provide a geometric interpretation: in the high-dimensional space of GAN features, each semantic class occupies a "hyper-cone" originating from the origin. If a pixel's feature vector falls inside that cone, it belongs to that class.
- Efficiency and Native Control: A 16-shot LSE achieves over 88% of the performance of a model trained on tens of thousands of images. Furthermore, using the LSE for editing works better than using external models because the LSE is "native" to the GAN's own feature space, leading to fewer visual artifacts.
Experimental Results
The study evaluated several architectures, including PGGAN, StyleGAN, and StyleGAN2, across datasets like CelebAHQ, FFHQ, and LSUN (Bedroom/Church).
Key quantitative results include:
- Accuracy: On the FFHQ dataset, the LSE achieved a mean Intersection over Union (mIoU) comparable to complex non-linear models within a narrow 3.5% margin.
- Few-Shot Performance: With just a single annotated image, the LSE achieved a plausible mIoU (69.8% for faces). By 16 images, it reached 88.1% of the full model's performance.
- Qualitative Editing: SIE successfully modified specific image components (like changing window positions or removing hair) while preserving the rest of the image's identity, outperforming standard color-stroke editing methods.
Limitations and Future Work
Despite its success, the model faces challenges with data imbalance. Rare semantic classes (like eyeglasses or hats) are harder for the LSE to represent if they appear infrequently in the training set, which can lead to mode collapse during editing. Additionally, the LSE is currently dependent on "teacher" segmentation models for its initial training labels.
Future research may explore whether this linear property exists in other generative models, such as Diffusion models or Variational Autoencoders, potentially offering new avenues for efficient and precise generative control.
Summary updated