Adversarial Dropout for Supervised and Semi-Supervised Learning

Sungrae Park, Jun-Keon Park, Su-Jin Shin, Il-Chul Moon

Introduction

Deep Neural Networks (DNNs) frequently struggle with overfitting, particularly in scenarios where labeled data is scarce. Traditional regularization techniques like Dropout mitigate this by randomly deactivating neurons to prevent co-adaptation. Conversely, Adversarial Training improves robustness by introducing "worst-case" perturbations to the input data. This paper introduces Adversarial Dropout (AdD), a novel approach that bridges these concepts by applying adversarial principles directly to the network's internal structure rather than the input space.

Methodology

Adversarial Dropout identifies and applies the "worst-case" dropout mask—the specific configuration of deactivated neurons that most confuses the network. By forcing the model to remain accurate even when its most informative internal pathways are blocked, AdD encourages the development of more robust and distributed representations.

The authors propose two primary variants:

  1. Supervised Adversarial Dropout (SAdD): Utilizes labels to find a mask that maximizes the divergence between the network's prediction and the ground truth.
  2. Virtual Adversarial Dropout (VAdD): Designed for semi-supervised learning, it identifies a mask that maximizes the difference between the outputs of a randomly dropped network and an adversarially dropped one.
Comparison of training paradigms: Self-ensembling (Pi-model), Adversarial Training, and the proposed Adversarial Dropout.

Finding the optimal adversarial mask is a complex combinatorial problem. To ensure computational efficiency during training, the authors employ a first-order Taylor expansion to linearize the loss function. This transformation allows the search for the adversarial mask to be treated as a 0/1 Knapsack Problem, which is efficiently solved using a greedy algorithm. This approach identifies neurons with the highest impact on the current prediction and "attacks" them by turning them off.

Algorithm 1: A greedy approach to find the adversarial dropout mask using a first-order Taylor expansion.

Experimental Evaluation

The effectiveness of AdD was evaluated on three standard benchmarks: MNIST, SVHN, and CIFAR-10. In semi-supervised settings, the models were trained with a limited number of labels (e.g., 1,000 for SVHN and 4,000 for CIFAR-10) to test generalization capabilities on the remaining unlabeled data.

Key Results and Insights

The experimental results demonstrate that AdD significantly improves performance, particularly in semi-supervised tasks. On the SVHN dataset with 1,000 labels, the combination of VAdD and Virtual Adversarial Training (VAT) achieved a 3.55% error rate, outperforming the $\Pi$-model (4.82%) and VAT alone (3.86%). Similarly, on CIFAR-10 with 4,000 labels, the VAdD + VAT combination reached a 9.22% error rate, a substantial improvement over the $\Pi$-model's 12.36%.

Beyond accuracy, the authors observed several qualitative benefits:

  • Targeting "Dead Filters": AdD effectively identifies neurons that may over-rely on specific training samples without contributing to generalizable features.
  • Structural Sparsity: The method results in more efficient neural activations and increased sparsity compared to standard random dropout.
  • Complementary Regularization: The study suggests that perturbing the network structure (AdD) and perturbing the input space (VAT) are complementary strategies that yield superior results when used together.

Conclusion

Adversarial Dropout offers a powerful new regularization tool by targeting internal structural vulnerabilities. While the method introduces some computational overhead due to Jacobian calculations and requires tuning of the constraint parameter $\delta$, its ability to learn sparse, robust representations makes it a compelling choice for both supervised and semi-supervised learning. Future research may explore the application of AdD to diverse architectures such as Transformers or Recurrent Neural Networks.

Summary updated

Abstract

Recently, the training with adversarial examples, which are generated by adding a small but worst-case perturbation on input examples, has been proved to improve generalization performance of neural networks. In contrast to the individually biased inputs to enhance the generality, this paper introduces adversarial dropout, which is a minimal set of dropouts that maximize the divergence between the outputs from the network with the dropouts and the training supervisions. The identified adversarial dropout are used to reconfigure the neural network to train, and we demonstrated that training on the reconfigured sub-network improves the generalization performance of supervised and semi-supervised learning tasks on MNIST and CIFAR-10. We analyzed the trained model to reason the performance improvement, and we found that adversarial dropout increases the sparsity of neural networks more than the standard dropout does.
BibTeX
@inproceedings{park2018adversarial,
  author = {Sungrae Park and Jun-Keon Park and Su-Jin Shin and Il-Chul Moon},
  title = {{Adversarial Dropout for Supervised and Semi-Supervised Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/aaai.v32i1.11634},
}
Figure