Semi-supervised learning based on generative adversarial network: a comparison between good GAN and bad GAN approach

Wenyuan Li, Zichen Wang, Jiayun Li, Jennifer Polson, William Speier, Corey W. ArnoldWorkshops @ CVPR 2019

TL;DR

This paper compares two GAN-based semi-supervised learning (SSL) approaches, "Good GAN" and "Bad GAN," on standard image datasets. They find that "Bad GAN" excels with more labeled data, achieving state-of-the-art results, while "Good GAN" is more robust to limited labeled data, especially when representative samples are carefully selected.

What Problem Does This Paper Address?

Semi-supervised learning (SSL) aims to leverage large amounts of unlabeled data to improve model performance, particularly when labeled data is scarce and expensive to obtain. While various SSL methods exist, this paper focuses on a specific challenge within GAN-based SSL: the incompatibility of loss functions between the generator and discriminator. This incompatibility can lead to generators producing "bad" or unrealistic images, which, paradoxically, can sometimes aid SSL. The paper investigates two distinct approaches that tackle this issue differently: "Bad GAN," which explicitly aims to generate "bad" samples to improve classifier decision boundaries, and "Good GAN" (Triple-GAN), which aims to generate high-quality, class-conditional samples to boost performance. The core problem addressed is understanding the trade-offs and performance characteristics of these two GAN-based SSL strategies across different data regimes.

Methods

The paper compares two GAN-based SSL methods:

  • Bad GAN: This approach builds upon the original GAN formulation for SSL by extending the discriminator to classify real data into its correct class and distinguish real from "bad" generated samples. The "bad" samples are generated by a generator that is trained with an objective function designed to avoid collapsing and to produce samples that lie in the complement of the data manifold. This is theorized to push the classifier's decision boundary between class manifolds, improving generalization.
  • Good GAN (Triple-GAN): This method addresses the generator-discriminator loss incompatibility by introducing a three-player game involving a generator (G), a discriminator (D), and a separate classifier (C). G and C generate pseudo-labels for unlabeled data, and D then distinguishes between real data-label pairs and generated data-label pairs. This approach aims for a global optimum where a good classifier implies a good generator and vice-versa, leveraging the REINFORCE algorithm to generate pseudo-labels for unlabeled data.

The authors use identical network architectures for the generator and classifier in both models to ensure a fair comparison.

Datasets

The paper evaluates the proposed methods on three widely adopted benchmark datasets:

  • MNIST: A dataset of handwritten digits (28x28 grayscale images).
  • SVHN (Street View House Numbers): A dataset of colored images of house numbers (32x32).
  • CIFAR10: A dataset of colored images across 10 general object classes (32x32).

For experiments, they varied the amount of labeled data, using specific counts like 20, 50, 100, 200 for MNIST; 500, 1000, 2000 for SVHN; and 1000, 2000, 4000, 8000 for CIFAR10. They also reserved a portion of the training data for validation.

Key Insights and Ideas

  • "Bad" samples can be beneficial: The paper validates the theoretical idea that "bad" or unrealistic generated samples, when strategically used, can improve classifier performance by forcing decision boundaries to lie between data manifolds.
  • Data regime matters: The choice between "Good GAN" and "Bad GAN" is highly dependent on the amount of labeled data available. "Bad GAN" shines with more data, while "Good GAN" is more resilient to scarcity.
  • Representative data selection is crucial for "Good GAN" with limited data: When labeled data is extremely scarce, the quality and representativeness of the selected samples significantly impact "Good GAN's" performance, more so than random selection.
  • Batch size is a critical hyperparameter: Both models are sensitive to batch size, but in different ways. "Bad GAN" benefits from moderate batch sizes to avoid generator collapse or loss of diversity, while "Good GAN" requires larger batch sizes for stable training due to its reliance on the REINFORCE algorithm.
  • Complementary strengths: The authors suggest that "Good GAN" and "Bad GAN" could be used in a complementary fashion, with "Good GAN" providing high-quality image-label pairs for training and "Bad GAN" refining decision boundaries.

Key Results

  • Classification Accuracy:
    • On MNIST, "Bad GAN" (ours) achieved up to 99.20% accuracy with 200 labeled samples, outperforming "Good GAN" (ours) which reached 99.17%. "Bad GAN" also matched the state-of-the-art "Bad GAN [3]" results.
  • On SVHN, "Bad GAN" (ours) achieved 95.47% with 2000 labeled samples, comparable to "Bad GAN [3]" and slightly better than "Good GAN" (ours) at 95.37%.
  • On CIFAR10, "Bad GAN" (ours) reached 85.47% with 8000 labeled samples, while "Good GAN" (ours) achieved 85.37%.
  • With low amounts of labeled data (e.g., 20 labeled samples on MNIST), "Good GAN" (ours) significantly outperformed "Bad GAN" (ours) (95.93% vs. 68.12%).
  • Generated Images: "Good GAN" generates clear, class-conditional images that exhibit disentangled styles and semantics. "Bad GAN" generates "bad" images that appear as fusions of different classes.
  • Batch Size Sensitivity: Figure 5 shows that "Bad GAN" generator loss is sensitive to batch size, with smaller batches leading to higher losses. Table 4 and Figure 6 illustrate that for "Bad GAN," performance degrades with very small or very large batch sizes. For "Good GAN," small batch sizes (e.g., 20, 50) lead to poor performance and even hurt the classifier, as shown in Table 5 and Figure 7.

Limitations and Future Work

  • Data Selection for "Good GAN": The paper highlights that for "Good GAN" with very limited labeled data, the selection of representative samples is critical. The performance can vary drastically based on this selection, as demonstrated in Figure 4. This suggests that more robust methods for sample selection or active learning could further improve "Good GAN."
  • Complementary Approaches: The authors propose that combining "Good GAN" and "Bad GAN" could lead to further performance improvements, suggesting future work in integrating their strengths.
  • Hyperparameter Sensitivity: While they followed existing work for most hyperparameters, they noted that "Bad GAN" is sensitive to batch size, which they investigated. Further exploration of hyperparameter tuning for both models could be beneficial.
  • Generalizability: While tested on standard datasets, the performance of these methods on more complex or domain-specific datasets might differ and warrants further investigation.

Summary updated

BibTeX
@inproceedings{li2019semi,
  author = {Wenyuan Li and Zichen Wang and Jiayun Li and Jennifer Polson and William Speier and Corey W. Arnold},
  title = {{Semi-supervised learning based on generative adversarial network: a comparison between good GAN and bad GAN approach}},
  booktitle = {Workshops @ CVPR},
  year = {2019},
}
Figure