Adversarial Dropout for Supervised and Semi-supervised Learning

Sungrae Park, Jun-Keon Park, Su-Jin Shin, Il-Chul Moon

Abstract

Recently, the training with adversarial examples, which are generated by adding a small but worst-case perturbation on input examples, has been proved to improve generalization performance of neural networks. In contrast to the individually biased inputs to enhance the generality, this paper introduces adversarial dropout, which is a minimal set of dropouts that maximize the divergence between the outputs from the network with the dropouts and the training supervisions. The identified adversarial dropout are used to reconfigure the neural network to train, and we demonstrated that training on the reconfigured sub-network improves the generalization performance of supervised and semi-supervised learning tasks on MNIST and CIFAR-10. We analyzed the trained model to reason the performance improvement, and we found that adversarial dropout increases the sparsity of neural networks more than the standard dropout does.

Introduction

Deep neural networks are notorious for overfitting—memorizing training data rather than learning generalizable patterns. To combat this, researchers commonly use dropout, which randomly "turns off" neurons during training to prevent them from becoming overly dependent on each other (co-adaptation). While effective, standard dropout is stochastic and "blind" to the model's current state.

Meanwhile, adversarial training improves robustness by adding small, "worst-case" perturbations to the input data. This paper explores applying the adversarial concept to the network structure itself. Instead of dropping neurons at random, the authors propose identifying and dropping the specific set of neurons that would most confuse the model, forcing it to learn more robust internal representations.

Methodology

The core contribution is the Adversarial Dropout (AdD) algorithm. Unlike standard dropout, which uses a random binary mask $\epsilon$, AdD searches for an adversarial mask $\epsilon^{adv}$ that maximizes the divergence between the model's standard output and its output with the mask applied.

The Adversarial Dropout framework compared to standard adversarial training and self-ensembling.

The authors propose two primary variants:

  1. Supervised Adversarial Dropout (SAdD): Maximizes the error relative to the true labels.
  2. Virtual Adversarial Dropout (VAdD): Maximizes the difference between the outputs of a randomly dropped network and an adversarially dropped one. This version is particularly powerful for semi-supervised learning because it does not require ground-truth labels for the adversarial step.

Finding the absolute "worst" mask is an integer programming problem. To make it practical, the authors use a fast approximation based on the first-order Taylor expansion of the loss function. This reduces the search to a variation of the 0/1 Knapsack Problem, which is solved efficiently using a greedy approach.

Experimental Results

The method was evaluated on three standard computer vision benchmarks: MNIST, SVHN, and CIFAR-10. The results demonstrate that Adversarial Dropout consistently outperforms standard dropout and other recent techniques like the $\Pi$-model:

  • Supervised Learning: On MNIST, SAdD achieved an error rate of 0.46%, outperforming standard dropout (0.53%). On CIFAR-10, VAdD reached 5.24% error, a significant improvement over the 5.81% of Virtual Adversarial Training (VAT).
  • Semi-Supervised Learning: The combination of VAdD and VAT achieved state-of-the-art performance. On SVHN with only 1,000 labels, they reached a 3.55% error rate. On CIFAR-10 with 4,000 labels, they achieved 9.22%, significantly lower than the 12.36% of the $\Pi$-model.

Analysis of Learned Representations

One of the most striking findings is that AdD forces "dead filters"—neurons that typically contribute very little—to become informative. By intentionally dropping the most "useful" neurons, the network is forced to find alternative pathways to solve the task, leading to more robust feature representations and increased sparsity.

Comparison of features learned by an autoencoder using standard dropout (left) vs. adversarial dropout (right). Note the reduction in "dead" or uninformative filters in the adversarial version.

Visualization of learned features shows that AdD produces cleaner, more standardized filters. While standard dropout might ignore neighborhoods of localized features, AdD preserves their characteristics while reducing noise.

Conclusion

Adversarial Dropout provides a novel way to regularize neural networks by targeting the internal structure rather than just the input data. While it introduces some computational overhead due to Jacobian calculations, the gains in generalization and sparsity are significant. Future work includes testing AdD on larger architectures like ResNets or Transformers and exploring adversarial perturbations for other network parameters beyond binary masks.

Summary updated

Abstract

Recently, the training with adversarial examples, which are generated by adding a small but worst-case perturbation on input examples, has been proved to improve generalization performance of neural networks. In contrast to the individually biased inputs to enhance the generality, this paper introduces adversarial dropout, which is a minimal set of dropouts that maximize the divergence between the outputs from the network with the dropouts and the training supervisions. The identified adversarial dropout are used to reconfigure the neural network to train, and we demonstrated that training on the reconfigured sub-network improves the generalization performance of supervised and semi-supervised learning tasks on MNIST and CIFAR-10. We analyzed the trained model to reason the performance improvement, and we found that adversarial dropout increases the sparsity of neural networks more than the standard dropout does.
BibTeX
@inproceedings{park2018adversarial,
  author = {Sungrae Park and Jun-Keon Park and Su-Jin Shin and Il-Chul Moon},
  title = {{Adversarial Dropout for Supervised and Semi-supervised Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/aaai.v32i1.11634},
}
Figure