Introduction
Deep Neural Networks (DNNs) frequently struggle with overfitting, particularly in scenarios where labeled data is scarce. Traditional regularization techniques like Dropout mitigate this by randomly deactivating neurons to prevent co-adaptation. Conversely, Adversarial Training improves robustness by introducing "worst-case" perturbations to the input data. This paper introduces Adversarial Dropout (AdD), a novel approach that bridges these concepts by applying adversarial principles directly to the network's internal structure rather than the input space.
Methodology
Adversarial Dropout identifies and applies the "worst-case" dropout mask—the specific configuration of deactivated neurons that most confuses the network. By forcing the model to remain accurate even when its most informative internal pathways are blocked, AdD encourages the development of more robust and distributed representations.
The authors propose two primary variants:
- Supervised Adversarial Dropout (SAdD): Utilizes labels to find a mask that maximizes the divergence between the network's prediction and the ground truth.
- Virtual Adversarial Dropout (VAdD): Designed for semi-supervised learning, it identifies a mask that maximizes the difference between the outputs of a randomly dropped network and an adversarially dropped one.
Fast Approximation via Greedy Search
Finding the optimal adversarial mask is a complex combinatorial problem. To ensure computational efficiency during training, the authors employ a first-order Taylor expansion to linearize the loss function. This transformation allows the search for the adversarial mask to be treated as a 0/1 Knapsack Problem, which is efficiently solved using a greedy algorithm. This approach identifies neurons with the highest impact on the current prediction and "attacks" them by turning them off.
Experimental Evaluation
The effectiveness of AdD was evaluated on three standard benchmarks: MNIST, SVHN, and CIFAR-10. In semi-supervised settings, the models were trained with a limited number of labels (e.g., 1,000 for SVHN and 4,000 for CIFAR-10) to test generalization capabilities on the remaining unlabeled data.
Key Results and Insights
The experimental results demonstrate that AdD significantly improves performance, particularly in semi-supervised tasks. On the SVHN dataset with 1,000 labels, the combination of VAdD and Virtual Adversarial Training (VAT) achieved a 3.55% error rate, outperforming the $\Pi$-model (4.82%) and VAT alone (3.86%). Similarly, on CIFAR-10 with 4,000 labels, the VAdD + VAT combination reached a 9.22% error rate, a substantial improvement over the $\Pi$-model's 12.36%.
Beyond accuracy, the authors observed several qualitative benefits:
- Targeting "Dead Filters": AdD effectively identifies neurons that may over-rely on specific training samples without contributing to generalizable features.
- Structural Sparsity: The method results in more efficient neural activations and increased sparsity compared to standard random dropout.
- Complementary Regularization: The study suggests that perturbing the network structure (AdD) and perturbing the input space (VAT) are complementary strategies that yield superior results when used together.
Conclusion
Adversarial Dropout offers a powerful new regularization tool by targeting internal structural vulnerabilities. While the method introduces some computational overhead due to Jacobian calculations and requires tuning of the constraint parameter $\delta$, its ability to learn sparse, robust representations makes it a compelling choice for both supervised and semi-supervised learning. Future research may explore the application of AdD to diverse architectures such as Transformers or Recurrent Neural Networks.
Summary updated