Repurposing GANs for One-shot Semantic Part Segmentation

Nontawat Tritrong, Pitchaporn Rewatbowornwong, Supasorn Suwajanakorn

TL;DR

This paper proposes a novel method to repurpose Generative Adversarial Networks (GANs) for few-shot semantic part segmentation. By extracting pixel-wise representations from a GAN trained on an object class, they achieve surprisingly good segmentation results with as little as one labeled example, outperforming traditional supervised methods that require significantly more data.

What Problem Does This Paper Address?

The core challenge this paper tackles is the data-hungry nature of semantic part segmentation. Traditionally, training models for this task requires a large number of meticulously annotated images, which are expensive and time-consuming to create. The authors are inspired by human learning, where a child can identify an object's parts after seeing just a few examples, leveraging prior knowledge. They question whether GANs, which excel at generating realistic images, implicitly learn meaningful structural information about object parts during their synthesis process. If so, this learned information could be leveraged for tasks like part segmentation with minimal supervision.

Methods

The paper introduces a two-pronged approach:

  1. Few-Shot Segmentation with GAN Representations:
    • Representation Extraction: A pre-trained GAN (specifically StyleGAN2) is used. For a given input image, they find a latent code z that, when fed to the GAN's generator, produces an image closely resembling the input. This is achieved through latent code optimization.
  2. Feature Generation: The generator's intermediate activation maps (from various layers) are extracted. These maps are spatially aligned and concatenated to form a rich, pixel-wise feature vector for each pixel in the input image. This vector is considered the "GAN-derived representation."
    • Segmentation Network Training: A simple segmentation network (CNN or MLP) is trained using these GAN-derived features and a small set of manually annotated images (the "one-shot" or "few-shot" examples). The network learns to map the GAN features to semantic part labels.
  3. Auto-Shot Segmentation (Extension):
    • Bypassing Latent Optimization: To address the computational cost of latent optimization during inference, they propose an "auto-shot" approach.
  4. Synthetic Dataset Generation: The trained GAN is used to generate a large dataset of synthetic images. The few-shot segmentation network (trained as above) is then used to generate segmentation masks for these synthetic images.
  5. Auto-Shot Segmenter Training: A new segmentation network (e.g., UNet) is trained on this large synthetic dataset. This network can then perform segmentation directly on real images without requiring GANs or latent optimization during inference.
  6. Data Augmentation: This auto-shot training incorporates extensive data augmentation (flips, scales, rotations, translations) to handle variations in object size and orientation, making it more robust for real-world scenarios.

Datasets

  • Training GANs: The paper mentions training GANs on datasets of the target class. For faces, they use CelebAMask-HQ. For cars and horses, they use images generated by pre-trained StyleGAN2 models (FFHQ, LSUN-Car, LSUN-Horse).
  • Evaluation:
    • Faces: CelebAMask-HQ.
  • Cars and Horses: PASCAL-Part dataset.

The paper does not introduce new benchmark datasets but rather leverages existing ones and synthetic data generated by GANs.

Key Insights and Ideas

  • GANs as Feature Extractors: The most significant insight is that GANs, primarily known for synthesis, implicitly learn powerful, semantically rich representations of object parts. These representations are "readily discriminative" for downstream tasks like segmentation.
  • One-Shot Learning Breakthrough: The ability to achieve competitive segmentation results with just one labeled example is a major advancement, drastically reducing annotation costs.
  • Auto-Shot Efficiency: The auto-shot extension effectively decouples the segmentation task from the GAN's inference process, making the approach much faster and more practical for real-world applications.
  • Middle Layers Matter: The study on GAN layer selection suggests that intermediate layers of the generator (controlling "middle style") are particularly effective for capturing features relevant to part segmentation, offering a nuanced understanding of GAN representations.
  • Logits vs. One-Hot Encoding: Using raw logit values (probabilities) instead of one-hot encoded labels for training the auto-shot segmenter leads to improved performance, indicating that preserving confidence scores is beneficial.

Key Results

  • Few-Shot Performance: The few-shot segmentation approach, using GAN-derived features, achieved comparable IOU scores to supervised baselines that used 10-50x more labeled data. For instance, their 1-shot segmenter on faces was comparable to a supervised baseline using ~150 labels, and their 10-shot segmenter matched a baseline using ~500 labels.
  • Auto-Shot Performance: The auto-shot segmenter achieved similar performance to the 10-shot segmenter, demonstrating the effectiveness of the synthetic data generation and training pipeline.
  • Cross-Domain Generalization: The method showed good results on diverse object classes like faces, cars, and horses, even when the GAN training data and real-world test images had variations in viewpoint and appearance.
  • Comparison to Other Representations: GAN-derived representations significantly outperformed those learned from VAEs, jigsaw solving, colorization, and HED for face segmentation, highlighting the unique advantage of GANs for this task.

Limitations and Future Work

  • GAN Dependency: The initial few-shot approach relies on a pre-trained GAN for the target class, which might not always be available or might require significant effort to train. The auto-shot method mitigates this during inference but still requires a trained GAN for data generation.
  • GAN Quality: The performance can be affected by the quality of GAN-generated images. Unrealistic or poorly formed images can lead to segmentation failures.
  • Domain Gap: While the auto-shot method improves generalization, there can still be a domain gap between GAN-generated data and real-world images, especially for complex or unusual scenarios.
  • Future Work: The authors suggest that this novel use of GANs for unsupervised representation learning can serve as a powerful "upstream" task for various problems requiring pixel-level understanding, including scene semantics and object part reasoning. They also explored segmentation of arbitrary shapes, indicating potential for more generalizable part segmentation.

Summary updated

Abstract

While GANs have shown success in realistic image generation, the idea of using GANs for other tasks unrelated to synthesis is underexplored. Do GANs learn meaningful structural parts of objects during their attempt to reproduce those objects? In this work, we test this hypothesis and propose a simple and effective approach based on GANs for semantic part segmentation that requires as few as one label example along with an unlabeled dataset. Our key idea is to leverage a trained GAN to extract pixel-wise representation from the input image and use it as feature vectors for a segmentation network. Our experiments demonstrate that GANs representation is "readily discriminative" and produces surprisingly good results that are comparable to those from supervised baselines trained with significantly more labels. We believe this novel repurposing of GANs underlies a new class of unsupervised representation learning that is applicable to many other tasks. More results are available at https://repurposegans.github.io/.
BibTeX
@inproceedings{tritrong2021repurposing,
  author = {Nontawat Tritrong and Pitchaporn Rewatbowornwong and Supasorn Suwajanakorn},
  title = {{Repurposing GANs for One-shot Semantic Part Segmentation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=rwPpcsfTpMD},
}
Figure