Abstract
While GANs have shown success in realistic image generation, the idea of using GANs for other tasks unrelated to synthesis is underexplored. Do GANs learn meaningful structural parts of objects during their attempt to reproduce those objects? In this work, we test this hypothesis and propose a simple and effective approach based on GANs for semantic part segmentation that requires as few as one label example along with an unlabeled dataset. Our key idea is to leverage a trained GAN to extract a pixel-wise representation from the input image and use it as feature vectors for a segmentation network. Our experiments demonstrate that this GAN-derived representation is "readily discriminative" and produces surprisingly good results that are comparable to those from supervised baselines trained with significantly more labels. We believe this novel repurposing of GANs underlies a new class of unsupervised representation learning, which can generalize to many other tasks.
Introduction
Semantic part segmentation—the task of labeling every pixel in an image according to the specific part of an object it belongs to—is notoriously data-hungry. Traditional supervised methods require thousands of manually annotated, pixel-perfect labels, which are expensive and time-consuming to produce.
While "few-shot" learning aims to solve this, most existing approaches still rely on "meta-learning" from other labeled datasets. This research asks a more fundamental question: Can a machine learn to identify parts of an object by simply looking at a large collection of unlabeled images? The authors hypothesize that for a GAN to successfully generate a realistic face or car, it must "understand" the underlying structure of those objects. This paper explores how to repurpose that "hidden" knowledge for segmentation with minimal human intervention.
Methodology
The proposed approach utilizes a two-stage pipeline to bridge the gap between generative modeling and discriminative segmentation.
- Representation Extraction: A GAN (specifically StyleGAN2) is pre-trained on a specific class (e.g., faces). To obtain features for a real image, the image is "projected" into the GAN’s latent space to find a code $z$ that reconstructs it. During the generation process, internal activation maps from every layer are collected, upsampled, and concatenated into a high-dimensional feature vector for every pixel.
- Few-shot Segmenter: A small network (an MLP or shallow CNN) is trained to map these GAN-derived feature vectors to semantic labels. Because the features are inherently descriptive, this network requires only 1 to 10 labeled images to achieve high accuracy.
- Auto-shot Segmentation: To address the latency of latent optimization and the "domain gap" between real and synthetic images, an "Auto-shot" model is introduced. The few-shot segmenter labels thousands of GAN-synthesized images, which are then used to train a standard U-Net with heavy data augmentation. This allows for real-time inference on real-world images without the GAN.
Datasets and Evaluation
The method was evaluated across three distinct object classes:
- Human Faces: Evaluated on CelebAMASK-HQ using a GAN trained on FFHQ.
- Cars: Evaluated on PASCAL-Part using a GAN trained on LSUN-Car.
- Horses: Evaluated on PASCAL-Part using a GAN trained on LSUN-Horse.
The GANs were trained entirely on unlabeled images, with segmentation labels reserved only for the few-shot training phase (1–10 images).
Key Results
The experiments demonstrate that GAN-derived representations are highly effective for segmentation tasks, often rivaling fully supervised models.
- Competitive Performance: On face segmentation, the 10-shot model achieved a weighted Intersection-over-Union (IOU) of 85.2, comparable to supervised baselines requiring over 500 labeled images.
- Efficiency: The 1-shot model outperformed a supervised baseline trained on 100 labels.
- Domain Generalization: The method successfully segments rigid objects (cars) and non-rigid objects (horses). While horse segmentation faced challenges due to GANs occasionally conflating riders with the animal, performance scaled significantly with additional labels.
Key Insights
- GANs as Unsupervised Feature Learners: GANs learn "readily discriminative" features. Even a linear classifier can produce decent segmentation masks from GAN features, proving that the generative process naturally organizes pixels into semantic groups.
- The "Computational Path" as a Descriptor: Every pixel in a GAN-generated image results from a specific path through the network's layers, serving as a unique "fingerprint" encoding local texture and global context.
- Synthetic Data Augmentation: By using the GAN to "hallucinate" a labeled dataset (Auto-shot), the authors create a robust model that handles varied scales and orientations better than the original GAN could.
Limitations and Future Work
The method's success is inherently tied to the quality of the underlying GAN; poor generation of specific parts leads to poor feature extraction. Additionally, latent projection remains computationally expensive, and the current focus on single-object classes limits application in complex, multi-object scenes. Future research may explore extending this framework to more diverse and cluttered environments.
Summary updated