A Scale Mixture Perspective of Multiplicative Noise in Neural Networks

Eric Nalisnick, Anima Anandkumar, Padhraic SmytharXiv 2015 · DOI · arXiv

Abstract

Corrupting the input and hidden layers of deep neural networks (DNNs) with multiplicative noise, often drawn from the Bernoulli distribution (or 'dropout'), provides regularization that has significantly contributed to deep learning's success. However, understanding how multiplicative corruptions prevent overfitting has been difficult due to the complexity of a DNN's functional form. In this paper, we show that when a Gaussian prior is placed on a DNN's weights, applying multiplicative noise induces a Gaussian scale mixture, which can be reparameterized to circumvent the problematic likelihood function. Analysis can then proceed by using a type-II maximum likelihood procedure to derive a closed-form expression revealing how regularization evolves as a function of the network's weights. Results show that multiplicative noise forces weights to become either sparse or invariant to rescaling. We find our analysis has implications for model compression as it naturally reveals a weight pruning rule that starkly contrasts with the commonly used signal-to-noise ratio (SNR). While the SNR prunes weights with large variances, seeing them as noisy, our approach recognizes their robustness and retains them. We empirically demonstrate our approach has a strong advantage over the SNR heuristic and is competitive to retraining with soft targets produced from a teacher model.

Introduction

Multiplicative noise techniques, most notably Dropout, are fundamental to the success of modern deep learning. While their effectiveness is well-documented, a rigorous theoretical understanding of their regularization properties in deep architectures remains elusive. In linear models, multiplicative noise acts as a data-dependent $L_2$ regularizer. However, in Deep Neural Networks (DNNs), the noise is processed through multiple non-linear layers, complicating direct analysis. This work introduces a new theoretical framework by treating multiplicative noise as a Gaussian Scale Mixture (GSM), providing a closed-form analytical expression for its regularization effect regardless of network depth or activation functions.

Methodology: From Noise to Hierarchical Models

The core of this approach lies in a Bayesian reinterpretation of multiplicative noise. By placing a Gaussian prior on the network weights, the product of the noise ($\lambda$) and the weight ($w$) defines a GSM. To make this mathematically tractable, the researchers employ a hierarchical reparameterization.

Instead of treating noise as a corruption during every forward pass, it is viewed as a hyperprior controlling the weight variance. This transformation results in an "identifiable" model where the noise is separated from the likelihood function, allowing for analysis via a type-II maximum likelihood procedure.

Graphical models comparing the unidentifiable multiplicative noise formulation (a) with the identifiable hierarchical reparameterization (b).

To handle the complexity of DNNs, the researchers use stochastic gradient Langevin dynamics to collect posterior samples of the weights. These samples provide the necessary moments (mean and variance) to calculate the derived regularization penalty.

The Dual Nature of Regularization and the SPR Heuristic

The derived penalty reveals that multiplicative noise forces weights into a binary state: they either become sparse (shrunk to zero) or invariant to rescaling. A weight survives if it is "robust" enough to remain useful despite the random scaling introduced during training.

This insight directly challenges the standard Signal-to-Noise Ratio (SNR) pruning rule ($|\mu|/\sigma$). In traditional signal processing, high variance ($\sigma$) is viewed as noise. Consequently, SNR-based pruning removes weights with high variances first. However, this paper argues that in noise-trained networks, a large posterior variance is a hallmark of quality and robustness. This leads to the proposed Signal-Plus-Robustness (SPR) heuristic ($|\mu| + \sigma$), which prioritizes retaining weights with either high means or high variances.

Experimental Results

The authors evaluated the SPR heuristic against the standard SNR rule and knowledge distillation (retraining with "soft targets") across three tasks: MNIST (digit classification), IMDB (sentiment analysis), and the Million Song Dataset (MSD, regression).

Performance of pruning heuristics (SNR vs. SPR) across MNIST, IMDB, and MSD datasets, along with a comparison of posterior weight moments.

As shown in the results, SPR consistently outperforms SNR. On the IMDB dataset, SPR allowed for the removal of 20% more parameters than SNR before performance degradation occurred. Furthermore, SPR-based pruning proved competitive with knowledge distillation, a significant result given that SPR is a simple post-hoc heuristic while distillation requires a full retraining cycle. Empirical validation of weight moments (Figure 4d) confirms that noise-trained networks develop a distinct population of robust, high-variance weights that are absent in models trained without noise.

Conclusion and Future Work

This scale mixture perspective provides both a theoretical foundation for multiplicative noise and a practical tool for model compression. While the current mathematical derivation strictly requires continuous noise distributions, the empirical success with Bernoulli noise suggests broader applicability. Future challenges include reducing the computational cost of calculating posterior moments for extremely large models and bridging the gap between the analytical framework and raw training accuracy improvements.

Summary updated

Abstract

Corrupting the input and hidden layers of deep neural networks (DNNs) with multiplicative noise, often drawn from the Bernoulli distribution (or 'dropout'), provides regularization that has significantly contributed to deep learning's success. However, understanding how multiplicative corruptions prevent overfitting has been difficult due to the complexity of a DNN's functional form. In this paper, we show that when a Gaussian prior is placed on a DNN's weights, applying multiplicative noise induces a Gaussian scale mixture, which can be reparameterized to circumvent the problematic likelihood function. Analysis can then proceed by using a type-II maximum likelihood procedure to derive a closed-form expression revealing how regularization evolves as a function of the network's weights. Results show that multiplicative noise forces weights to become either sparse or invariant to rescaling. We find our analysis has implications for model compression as it naturally reveals a weight pruning rule that starkly contrasts with the commonly used signal-to-noise ratio (SNR). While the SNR prunes weights with large variances, seeing them as noisy, our approach recognizes their robustness and retains them. We empirically demonstrate our approach has a strong advantage over the SNR heuristic and is competitive to retraining with soft targets produced from a teacher model.
BibTeX
@misc{nalisnick2015scale,
  author = {Eric Nalisnick and Anima Anandkumar and Padhraic Smyth},
  title = {{A Scale Mixture Perspective of Multiplicative Noise in Neural Networks}},
  year = {2015},
  eprint = {1506.03208},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1506.03208},
}
Figure