Bayesian Dropout

Tue Herlau, Morten Mørup, Mikkel N. Schmidt

Abstract

Dropout has recently emerged as a powerful and simple method for training neural networks preventing co-adaptation by stochastically omitting neurons. Dropout is currently not grounded in explicit modelling assumptions which so far has precluded its adoption in Bayesian modelling. Using Bayesian entropic reasoning we show that dropout can be interpreted as optimal inference under constraints. We demonstrate this on an analytically tractable regression model providing a Bayesian interpretation of its mechanism for regularizing and preventing co-adaptation as well as its connection to other Bayesian techniques. We also discuss two general approximate techniques for applying Bayesian dropout for general models, one based on an analytical approximation and the other on stochastic variational techniques. These techniques are then applied to a Baysian logistic regression problem and are shown to improve performance as the model become more misspecified. Our framework roots dropout as a theoretically justified and practical tool for statistical modelling allowing Bayesians to tap into the benefits of dropout training.

Introduction

In machine learning, co-adaptation occurs when a model relies on highly specific, complex combinations of features to make predictions. While effective on training data, this often fails on novel data because these combinations may represent noise rather than signal. Standard Bayesian methods, while powerful, assume perfect model specification. When a model is misspecified, the Bayesian posterior can become overconfident in incorrect parameters.

Dropout—the stochastic omission of neurons during training—was developed to mitigate co-adaptation in neural networks. However, its status as an algorithmic heuristic has historically limited its application within formal Bayesian frameworks. This paper bridges that gap by demonstrating that Dropout is a mathematically optimal form of inference under specific robustness constraints.

Methods: Dropout as Maximum Entropy Inference

The authors utilize the Extended Method of Maximum Entropy (ME) to ground Dropout in first principles. Unlike standard Bayesian inference, which updates a prior solely based on data, ME allows for updates that satisfy specific constraints.

  1. The Dropout Constraint: Bayesian Dropout is defined by imposing a requirement that the model remains robust even when parts of it are missing. Specifically, the probability of a weight being "dropped" ($f$) is fixed and cannot be learned from the data.
  2. The Variational Problem: The goal is to find the distribution with the highest entropy (least bias) that satisfies both the observed data and the dropout constraint.
  3. The Resulting Posterior: The derived posterior uses the expected log-likelihood across all possible "dropped out" versions of the model, rather than a standard likelihood.

For linear regression, this approach yields a closed-form solution where Dropout acts as a data-dependent regularizer, pulling the model toward solutions where each feature can contribute independently.

Empirical Results

Shrinkage and Robustness

The framework was tested on QSPR (Quantitative Structure-Property Relationship) data and synthetic datasets. As shown in Figure 5, Dropout's regularization differs significantly from Ridge ($L_2$) and Lasso ($L_1$). While Ridge regression may assign features opposite signs to "cancel each other out" (a form of co-adaptation), Bayesian Dropout forces consistent, independent signs.

Comparison of shrinkage patterns (left) and performance curves under misspecification (right) for Ridge regression and Dropout.

Furthermore, synthetic tests revealed that while standard Bayesian Ridge regression is optimal for perfectly specified models, it is fragile. Under "covariate shift" (where input distributions change) or in the presence of "junk features," Bayesian Dropout maintains significantly more stable performance.

Real-World Performance: Amazon Review Dataset

The authors evaluated the model on a large-scale Amazon Review dataset across four categories: Books, DVDs, Electronics, and Kitchen. Using high-dimensional "bag-of-words" features, Bayesian Dropout consistently outperformed standard Bayesian linear regression.

Classification error across Amazon Review categories, comparing Ridge regression and Bayesian Dropout across varying regularization strengths.

As illustrated in Figure 6, Bayesian Dropout provides a broader "sweet spot" for hyperparameters and achieves lower overall error, demonstrating superior generalization in practical, high-dimensional scenarios.

Conclusion

This work establishes Dropout as a theoretically justified tool for statistical modeling. By framing it as optimal inference under constraints, the authors allow Bayesian practitioners to leverage Dropout's benefits for handling model misspecification and preventing co-adaptation. While this study focuses on linear models for analytical tractability, the framework provides a foundation for applying Bayesian Dropout to more complex, non-linear architectures through stochastic variational techniques.

Summary updated

Abstract

Dropout has recently emerged as a powerful and simple method for training neural networks preventing co-adaptation by stochastically omitting neurons. Dropout is currently not grounded in explicit modelling assumptions which so far has precluded its adoption in Bayesian modelling. Using Bayesian entropic reasoning we show that dropout can be interpreted as optimal inference under constraints. We demonstrate this on an analytically tractable regression model providing a Bayesian interpretation of its mechanism for regularizing and preventing co-adaptation as well as its connection to other Bayesian techniques. We also discuss two general approximate techniques for applying Bayesian dropout for general models, one based on an analytical approximation and the other on stochastic variational techniques. These techniques are then applied to a Baysian logistic regression problem and are shown to improve performance as the model become more misspecified. Our framework roots dropout as a theoretically justified and practical tool for statistical modelling allowing Bayesians to tap into the benefits of dropout training.
BibTeX
@inproceedings{herlau2022bayesian,
  author = {Tue Herlau and Morten Mørup and Mikkel N. Schmidt},
  title = {{Bayesian Dropout}},
  booktitle = {ANT/EDI40},
  year = {2022},
  doi = {10.1016/j.procs.2022.03.105},
}
Figure