MarginGAN: Adversarial Training in Semi-Supervised Learning

Jinhao Dong, Tong LinNeurIPS 2019

Introduction

Semi-supervised learning (SSL) aims to build powerful models using a small amount of labeled data and a vast pool of unlabeled data. A common strategy in this domain is pseudo-labeling, where the model "guesses" the labels for unlabeled points and treats those guesses as ground truth.

However, pseudo-labeling has a major flaw: if the model makes a wrong guess, it reinforces its own mistake, leading to poor generalization. While Generative Adversarial Networks (GANs) have been used to help by "filling in" the data distribution, previous methods often struggled to effectively use generated "fake" data to refine the actual decision boundaries of the classifier.

MarginGAN is a novel SSL framework that tackles the challenge of making models more robust to incorrect pseudo-labels by incorporating margin theory into the adversarial process. By forcing the generator to produce "hard" examples near decision boundaries and training the classifier to maximize margins on real data while minimizing them on fakes, the model achieves state-of-the-art accuracy and significantly faster training times.

Methods: A Three-Player Adversarial Game

The authors propose a unique three-component architecture that moves beyond the traditional two-player GAN setup:

  1. The Discriminator ($D$): Functions like a standard GAN discriminator, distinguishing between real (labeled/unlabeled) data and fake data from the generator.
  2. The Classifier ($C$): This is the heart of the innovation. It is trained to:
  3. Increase the margin for real and unlabeled data (using pseudo-labels). A large margin means the classifier is very confident and the data point is far from the decision boundary.
  4. Decrease the margin for fake data using a new "Inverted Cross Entropy" (ICE) loss. This forces the classifier to be uncertain about generated examples.
  5. The Generator ($G$): It plays a double game. It tries to fool the discriminator (by looking realistic) and the classifier (by producing examples with large margins).
MarginGAN Architecture

This setup creates a "push-and-pull" effect. To fool the classifier, the generator must produce fake examples that look like they belong to a specific class but are actually located near the "correct" decision boundaries. These "informative" fakes act like support vectors in a Support Vector Machine (SVM), helping the classifier refine its boundaries even when some pseudo-labels are noisy.

Key Insights and Training Dynamics

MarginGAN introduces several counter-intuitive but highly effective dynamics to the training process:

  • The "Bad" Generator is a Good Teacher: A generator that produces "perfect" images isn't actually very helpful for classification. MarginGAN succeeds by encouraging the generator to produce "bad" or "hard" examples—images that sit right on the edge of class boundaries.
  • Adversarial Margin Training: By explicitly minimizing the margin of fake data, the classifier learns to ignore "bogus" high-confidence regions, making it much more tolerant of the errors inherent in pseudo-labeling.
  • Synergy and Speed: One of the most surprising findings was that adding the generator actually decreased training time. The generated samples provide a more immediate and "informative" signal for where the boundaries should be, allowing the model to converge faster.

Experimental Results

The model was evaluated on three standard computer vision benchmarks for SSL: MNIST (handwritten digits), SVHN (Street View House Numbers), and CIFAR-10 (natural images). The authors tested these under various "label-scarce" settings.

MarginGAN demonstrated significant improvements over existing state-of-the-art methods like Triple-GAN and Mean Teacher:

  • MNIST: With only 100 labels, MarginGAN achieved an error rate of 3.53%, outperforming traditional pseudo-labeling methods which typically hovered around 10-16% error.
  • CIFAR-10: With 4,000 labels, it achieved a 6.44% error rate, a massive improvement over the "Mean Teacher" baseline of 13.20%.
Training Time and Extreme Ablation Results

Furthermore, the framework proved incredibly robust. In an "extreme" ablation study where the classifier was intentionally fed only unlabeled data (no labels at all after pre-training), the error rate for standard training exploded to 89.53%. However, when the generator was added ($U+G$), the error rate stayed at a highly respectable 7.40%. This proves the generator's ability to stabilize training against incorrect pseudo-labels. Additionally, as shown in the results, adding the generator to the labeled and unlabeled mix ($L+U+G$) reduced training time by over 70% compared to using unlabeled data alone ($L+U$).

Limitations and Future Work

The authors note a potential pitfall: overfitting. As the generator becomes more realistic over time, the "gap" between real and fake data shrinks. If the generator becomes too good, the classifier might start reducing margins on samples that actually look real, which can eventually hurt performance.

Future work could explore more sophisticated ways to balance the "realism" of the generator with its "informativeness" to ensure the classifier continues to learn effectively throughout the entire training process. Testing the framework on non-image data or more complex, high-resolution datasets would also be a logical next step.

Summary updated

BibTeX
@inproceedings{dong2019margingan,
  author = {Jinhao Dong and Tong Lin},
  title = {{MarginGAN: Adversarial Training in Semi-Supervised Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
}
Figure