Advanced Dropout: A Model-free Methodology for Bayesian Dropout Optimization

Jiyang Xie, Zhanyu Ma, Guoqiang Zhang, Jing-Hao Xue, Zheng-Hua Tan, Jun GuoIEEE TPAMI 2020 · DOI · OpenReview

TL;DR

This paper introduces "Advanced Dropout," a novel, model-free method for optimizing dropout rates in deep neural networks. By employing a flexible distribution and a parametric prior, it adaptively adjusts dropout rates based on input features, leading to significant improvements in preventing overfitting and boosting performance across various computer vision tasks and beyond.

What Problem Does This Paper Address?

Deep neural networks (DNNs) have achieved remarkable success, but their vast number of parameters makes them prone to overfitting, especially when training data is limited. Dropout, a popular regularization technique, addresses this by randomly dropping nodes during training to prevent co-adaptation. However, existing dropout methods often rely on fixed dropout rates or assume specific distributions for dropout masks, which can introduce biases and limit their effectiveness. The authors aim to overcome these limitations by proposing a more flexible and adaptive dropout approach that can generalize across existing methods and improve performance.

Methods

The core of the proposed "Advanced Dropout" lies in two key components:

  • Model-Free Distribution: Instead of relying on pre-defined distributions like Bernoulli or Gaussian for dropout masks, Advanced Dropout uses a flexible, model-free distribution. This distribution can approximate various existing dropout distributions (e.g., Bernoulli, Gaussian, uniform, beta) by adjusting its parameters, offering greater adaptability.
  • Parametric Prior and SGVB Optimization: The parameters of this model-free distribution are adaptively adjusted based on input features using a parametric prior. This prior is optimized end-to-end with the DNN using Stochastic Gradient Variational Bayes (SGVB) inference, allowing for dynamic dropout rate adjustments during training.

This approach allows the dropout rate to be learned and adapted based on the input data, rather than being fixed or determined by a rigid distribution.

Datasets

The authors evaluated their method on a range of datasets:

  • Image Classification: MNIST, CIFAR-10, CIFAR-100, miniImageNet, Caltech-256, ImageNet-32x32, and ImageNet.
  • Text Classification: Reuters-21578.
  • Regression: Four UCI datasets (Boston Housing, Concrete Strength, Wine Quality Red, Yacht Hydrodynamics).

Key Insights and Ideas

  • Flexibility is Key: The paper highlights that a model-free distribution for dropout masks, capable of approximating various existing distributions, offers superior adaptability compared to fixed or model-specific distributions.
  • Adaptive Dropout Rates Improve Performance: By allowing dropout rates to be dynamically adjusted based on input features, the model can better prevent overfitting and improve generalization.
  • SGVB is Crucial for End-to-End Learning: Integrating the parametric prior optimization with SGVB inference enables seamless end-to-end training, making the adaptive dropout effective.
  • Beyond Image Classification: The research demonstrates that Advanced Dropout's benefits extend beyond image tasks, showing strong performance in text classification and regression.

Key Results

  • Superior Accuracy: Advanced Dropout consistently outperformed nine other dropout techniques across all seven image classification datasets, achieving the best accuracy in most cases. For instance, on MNIST, it achieved 98.89% accuracy, outperforming the next best by a small margin but significantly improving over no dropout.
  • Effective Overfitting Prevention: Ablation studies and experiments with reduced training data or increased model depth showed that Advanced Dropout significantly reduces the gap between training and test accuracies, indicating strong overfitting prevention capabilities.
  • Competitive Training Time: While not the fastest, Advanced Dropout's training time per epoch was comparable to other advanced dropout variants, suggesting that the overhead of adaptive learning is manageable.
  • Improved Uncertainty Inference and Network Pruning: The method also showed promising results when extended to uncertainty inference and network pruning tasks, outperforming existing methods.

Limitations and Future Work

  • Computational Overhead: While competitive, the adaptive nature of Advanced Dropout does introduce some computational overhead compared to standard dropout.
  • Further Exploration of Distributions: The paper suggests that exploring other seed distributions and mapping functions for the model-free distribution could lead to further improvements.
  • Broader Applications: The authors successfully extended the method to uncertainty inference, network pruning, text classification, and regression, suggesting potential for wider adoption in various deep learning applications.

Summary updated

Abstract

Due to lack of data, overfitting ubiquitously exists in real-world applications of deep neural networks (DNNs). We propose advanced dropout, a model-free methodology, to mitigate overfitting and improve the performance of DNNs. The advanced dropout technique applies a model-free and easily implemented distribution with parametric prior, and adaptively adjusts dropout rate. Specifically, the distribution parameters are optimized by stochastic gradient variational Bayes in order to carry out an end-to-end training. We evaluate the effectiveness of the advanced dropout against nine dropout techniques on seven computer vision datasets (five small-scale datasets and two large-scale datasets) with various base models. The advanced dropout outperforms all the referred techniques on all the datasets.We further compare the effectiveness ratios and find that advanced dropout achieves the highest one on most cases. Next, we conduct a set of analysis of dropout rate characteristics, including convergence of the adaptive dropout rate, the learned distributions of dropout masks, and a comparison with dropout rate generation without an explicit distribution. In addition, the ability of overfitting prevention is evaluated and confirmed. Finally, we extend the application of the advanced dropout to uncertainty inference, network pruning, text classification, and regression. The proposed advanced dropout is also superior to the corresponding referred methods. Codes are available at https://github.com/PRIS-CV/AdvancedDropout.
BibTeX
@article{xie2020advanced,
  author = {Jiyang Xie and Zhanyu Ma and Guoqiang Zhang and Jing-Hao Xue and Zheng-Hua Tan and Jun Guo},
  title = {{Advanced Dropout: A Model-free Methodology for Bayesian Dropout Optimization}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2020},
  doi = {10.1109/tpami.2021.3083089},
}
Figure