FX-GAN: Self-Supervised GAN Learning via Feature Exchange

Rui Huang, Wenju Xu, Teng-Yok Lee, Anoop Cherian, Ye Wang, Tim MarksWACV 2020

TL;DR

This paper introduces FX-GAN, a novel self-supervised learning technique for Generative Adversarial Networks (GANs). By training the discriminator to identify structurally corrupted real images (via feature exchange), FX-GAN guides the generator to produce more realistic and structurally coherent images, leading to improved generation quality and training stability.

What Problem Does This Paper Address?

Generative Adversarial Networks (GANs) are powerful tools for generating realistic images, but training them can be notoriously unstable and challenging. A key difficulty lies in ensuring that generated images possess not only realistic textures but also correct structural composition (e.g., the correct arrangement of facial features in a face). While standard GAN discriminators are trained to distinguish real from fake images, they often struggle to learn these higher-order structural properties. Existing self-supervised methods, like rotation prediction, help, but they might not fully capture local structural consistency. This paper aims to improve GAN training by introducing a self-supervised task that specifically forces the discriminator to learn and enforce structural consistency within real images, thereby guiding the generator towards producing more structurally sound outputs.

Methods

The core innovation of FX-GAN is the "feature exchange" operation and a new self-supervised task for the discriminator.

  • Feature Exchange Operator: This operator takes a feature map from an intermediate layer of the discriminator and randomly swaps feature vectors at two spatial locations. This creates a "structurally corrupted" version of a real image's feature representation.
  • Dual-Task Discriminator: The discriminator is modified to perform two tasks simultaneously:
  • Real vs. Fake Classification: The standard GAN task of distinguishing generated images from real ones.
  • Exchanged vs. Original Classification: An auxiliary task where the discriminator must identify whether a real image's feature map has undergone the feature exchange operation.
  • Loss Function: A combined loss function trains the discriminator on both tasks. The generator is then trained to fool the discriminator on the real vs. fake task, implicitly benefiting from the discriminator's improved understanding of structural consistency.

The authors hypothesize that by forcing the discriminator to detect subtle structural inconsistencies introduced by feature exchange, it learns a richer representation of "correct" image structure. This improved discriminator then provides a stronger learning signal to the generator.

Datasets

The paper evaluates FX-GAN on the following datasets:

  • Unconditional Generation:
    • CelebA-HQ
    • LSUN-bedroom
    • CIFAR10
  • Conditional Generation:
    • ImageNet

Key Insights and Ideas

  • Structural Consistency is Key: The paper highlights that generating realistic images requires more than just good texture; correct structural composition is crucial and often harder for GANs to learn.
  • Discriminator as a Structural Inspector: By training the discriminator to spot "structurally corrupted" real images, we implicitly teach it what "structurally sound" looks like. This is a more direct way to enforce structure than global tasks like rotation prediction.
  • Local Feature Exchange, Global Impact: Swapping small blocks of features (1x1 block size) in intermediate feature maps proves most effective. This suggests that even localized "corruption" can reveal global structural weaknesses, and the discriminator learns to identify these.
  • Synergy with Existing GANs: FX-GAN is designed to be a plug-in module that can enhance existing GAN architectures and self-supervised methods, demonstrating its versatility.

Key Results

  • Improved Image Quality: FX-GAN consistently achieves lower Fréchet Inception Distance (FID) scores across multiple datasets (CelebA-HQ, LSUN-bedroom, CIFAR10) compared to baseline DG-SNGAN and DG-SNGAN with rotation-based self-supervision (SS-Rot). For instance, on LSUN-bedroom, FX-GAN achieved an FID of 12.90, significantly better than SS-Rot's 24.21.
  • Enhanced Training Stability: The self-supervision signal from feature exchange helps stabilize GAN training, preventing divergence issues observed in baseline models.
  • Better Performance on Conditional GANs: On ImageNet, FX-GAN improved both the Inception Score and FID for both DG-SNGAN and SAGAN baselines, indicating better generation quality and diversity.
  • Qualitative Improvements: Visual examples show that FX-GAN generates images with more realistic scene layouts and local details, and exhibits greater diversity.
  • Minor Computational Overhead: The feature exchange operation adds only a ~20% increase in training time per iteration.

Limitations and Future Work

  • Hyperparameter Sensitivity: The effectiveness of FX-GAN is correlated with the difficulty of the "Exchanged?" task. The paper explores block size and feature map size, suggesting smaller blocks in larger feature maps are optimal, but further tuning might be needed.
  • Synergy with Self-Attention: The paper notes that combining FX with self-attention (as in SAGAN) can sometimes lead to unstable training or less pronounced improvements, suggesting that their synergy needs further investigation. Future work could explore how to best combine these powerful techniques.
  • Potential for Artifacts: While feature exchange in feature space avoids raw pixel artifacts, the paper acknowledges that applying it directly to raw pixels is prone to issues. The choice of feature map and block size is crucial to avoid making the task too easy or too difficult.

Summary updated

BibTeX
@inproceedings{huang2020fx,
  author = {Rui Huang and Wenju Xu and Teng-Yok Lee and Anoop Cherian and Ye Wang and Tim Marks},
  title = {{FX-GAN: Self-Supervised GAN Learning via Feature Exchange}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure