Learning Fast Converging, Effective Conditional Generative Adversarial Networks With a Mirrored Auxiliary Classifier

Zi WangWACV 2021

TL;DR

This paper introduces MAC-GAN, a novel conditional Generative Adversarial Network (GAN) architecture that improves training stability and image generation quality. By modifying the discriminator's auxiliary classifier to include separate "real" and "fake" nodes for each class, MAC-GAN effectively disentangles real and generated samples, leading to faster convergence and higher fidelity image synthesis compared to existing methods.

What Problem Does This Paper Address?

Generative Adversarial Networks (GANs) are powerful tools for generating realistic images, but training them, especially conditional GANs (cGANs) that generate images of specific classes, can be notoriously difficult. Standard cGANs struggle with convergence due to the inherent gap between real and generated data distributions. Existing solutions like AC-GAN (Auxiliary Classifier GAN) and FC-GAN (Fast Converging GAN) attempt to address this by incorporating an auxiliary classifier within the discriminator. AC-GAN uses a single classifier for real/fake and class prediction, while FC-GAN introduces an additional "fake" class to simplify the discriminator's task. However, FC-GAN's single "fake" class can become a bottleneck as generated samples become more diverse, and AC-GAN can still face training challenges. This paper aims to bridge this gap by proposing a more robust auxiliary classifier that can handle the complexities of multi-modal generated distributions more effectively, leading to faster convergence and better image quality.

Methods

The core innovation of MAC-GAN lies in its Mirrored Auxiliary Classifier (MAC) within the discriminator. Unlike previous approaches, MAC-GAN's auxiliary classifier is designed with two nodes per class: one for "real" samples of that class and one for "fake" samples of that class. This "mirrored" structure allows the discriminator to explicitly distinguish between real and generated images within the same class.

The objective function is a combination of:

  • Source Loss: Standard GAN loss to distinguish real from fake.
  • Classification Loss: The MAC-GAN's auxiliary classifier loss, which now assigns real samples to their correct "real class" node and generated samples to their corresponding "fake class" node.

This design choice is motivated by the observation that while real samples for a given class might have a relatively stable distribution, generated samples (especially at earlier training stages) can exhibit multimodal distributions. By providing separate nodes for real and fake within each class, MAC-GAN avoids confusing the classifier with a single "fake" class that has to encompass all generated variations.

Datasets

The proposed MAC-GAN was evaluated on three widely used benchmark datasets:

  • MNIST: A dataset of handwritten digits (28x28 resolution).
  • CIFAR-10: A dataset of 10 classes of small images (32x32 resolution).
  • ILSVRC-2012 (ImageNet): A large-scale dataset with 1,000 classes of images (cropped to 224x224 for experiments).

Key Insights and Ideas

  • Disentangling Real and Fake within Classes is Crucial: The paper highlights that simply assigning all generated samples to a single "fake" class (as in FC-GAN) or relying on a single classifier for both source and class (as in AC-GAN) can hinder convergence. Explicitly separating real and fake within each class provides a more granular and effective signal for the discriminator.
  • Faster Convergence, Better Quality: The mirrored auxiliary classifier leads to significantly faster convergence of both the generator and discriminator, which in turn results in higher quality image generation, even at earlier training stages.
  • Robustness to Multimodal Distributions: The MAC-GAN's architecture is particularly effective at handling the multimodal nature of generated samples, a common challenge in GAN training.
  • Potential for Transfer Learning: The improved classification performance of the auxiliary classifier in MAC-GAN suggests its utility in transfer learning tasks.

Key Results

  • Visual Fidelity: MAC-GAN generates recognizable samples much earlier than AC-GAN and produces the best visual fidelity on both MNIST and CIFAR-10, with clearer images and better diversity.
  • Loss Convergence: MAC-GAN converges to a Nash equilibrium significantly faster than AC-GAN on MNIST and CIFAR-10. While FC-GAN also shows fast convergence, MAC-GAN achieves comparable or better results.
  • Quantitative Metrics:
    • Parzen Window Estimate: MAC-GAN consistently achieved higher scores on MNIST and CIFAR-10, indicating better generated sample quality. For example, on CIFAR-10 after 300 epochs, MAC-GAN reached 684.9, a 11.3% improvement over AC-GAN and FC-GAN.
  • Inception Score: MAC-GAN demonstrated strong performance, outperforming AC-GAN and FC-GAN on MNIST and achieving competitive results on CIFAR-10 and ILSVRC-2012.
  • FID Score (CIFAR-10): MAC-GAN achieved lower FID scores than AC-GAN and FC-GAN, indicating generated images are closer to real images in feature space.
  • Auxiliary Classifier Accuracy: MAC-GAN's auxiliary classifier achieved the highest test accuracy on ILSVRC-2012 (71.80%) and comparable performance to AC-GAN on MNIST and CIFAR-10, outperforming FC-GAN.

Limitations and Future Work

  • Limitations: The paper notes that for the challenging ILSVRC-2012 dataset, none of the models (AC-GAN, FC-GAN, MAC-GAN) fully converge to a Nash equilibrium, highlighting the inherent difficulty of training GANs on such large-scale datasets.
  • Future Work:
    • Fusion of Models: The authors suggest exploring a conditional GAN that fuses the objective functions of AC-GAN, FC-GAN, and MAC-GAN to potentially achieve even faster convergence and robustness across different training stages.
  • Adversarial Loss Removal: Given that MAC-GAN's auxiliary classifier already handles real/fake distinctions within classes, the authors hypothesize that the adversarial loss might be removable, potentially further stabilizing GAN training.
  • Applications in Computer Vision: The paper proposes extending MAC-GAN to other challenging computer vision tasks.

Summary updated

BibTeX
@inproceedings{wang2021learning,
  author = {Zi Wang},
  title = {{Learning Fast Converging, Effective Conditional Generative Adversarial Networks With a Mirrored Auxiliary Classifier}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
}
Figure