Uncertainty Estimation via Stochastic Batch Normalization

Andrei Atanov, Arsenii Ashukha, Dmitry Molchanov, Kirill Neklyudov, Dmitry P. Vetrov

Abstract

Deep Neural Networks (DNNs) are notoriously overconfident, often assigning high probabilities to incorrect predictions, particularly when encountering out-of-distribution (OOD) data. While techniques like Bayesian Neural Networks, Dropout, and Deep Ensembles attempt to address this, they frequently suffer from computational intractability, performance degradation in modern architectures like ResNets, or high memory overhead. We introduce Stochastic Batch Normalization (SBN), a method that leverages the inherent stochasticity of Batch Normalization (BN) statistics. By treating BN means and variances as random variables during inference, SBN provides an efficient and scalable uncertainty estimation framework. Our evaluation across multiple architectures shows that SBN improves OOD detection and model calibration while preserving predictive accuracy.

1. Introduction

In safety-critical fields such as medical diagnosis or autonomous driving, knowing when a model is uncertain is as vital as the prediction itself. Standard Batch Normalization is a staple of modern DNNs, but it is "inconsistent": it is stochastic during training (depending on mini-batch samples) but deterministic at test time (using fixed running averages). SBN restores this consistency by modeling the distribution of these statistics, allowing for Monte Carlo uncertainty estimation without requiring architectural changes or training multiple models.

2. Methodology

SBN is based on a probabilistic interpretation of Batch Normalization, where training is viewed as a form of approximate Bayesian inference.

2.1 Parametric Approximation

To make inference efficient, the authors model the distributions of the batch statistics. Empirical analysis suggests that the batch mean ($\mu$) follows a Gaussian distribution, while the batch variance ($\sigma^2$) follows a Log-Normal distribution.

Empirical distributions of batch statistics $\mu$ and $\sigma^2$ across different layers, justifying the choice of Gaussian and Log-Normal parametric approximations.

2.2 Inference Procedure

During training, SBN uses exponential smoothing to estimate the parameters of these distributions (e.g., the variance of the means). At test time, SBN performs multiple forward passes, sampling new $\mu$ and $\sigma^2$ values for each layer from these learned distributions. This allows for "free" uncertainty estimation from pre-trained BN networks.

3. Experimental Results

The researchers evaluated SBN using standard benchmarks, including MNIST (OOD: notMNIST) and CIFAR-10 (OOD: CIFAR5), across LeNet-5, VGG-11, and ResNet-18 architectures.

3.1 Out-of-Distribution Detection

SBN consistently produces higher entropy for OOD images compared to standard BN. Furthermore, SBN exhibits strong synergy with existing methods. As shown in the entropy Cumulative Distribution Function (CDF) plots, adding SBN to Dropout or Deep Ensembles (dashed lines) shifts the distribution toward higher entropy, indicating improved uncertainty quantification.

Entropy CDFs for LeNet-5, VGG-11, and ResNet-18. The dashed lines demonstrate how SBN enhances the uncertainty estimation of Dropout and Deep Ensembles by shifting the CDF toward higher entropy values.

3.2 Calibration and Accuracy

SBN significantly improves model calibration. For example, on ResNet-18 (CIFAR5), SBN improved the Negative Log-Likelihood (NLL) of a Deep Ensemble from 0.138 to 0.110. Crucially, SBN maintains high predictive accuracy; in some cases, such as LeNet-5, the test error even decreased from 0.51% (with Dropout) to 0.49% (with SBN).

4. Conclusion

Stochastic Batch Normalization offers a scalable, ResNet-friendly alternative for uncertainty estimation. By leveraging the stochasticity already present in the BN layers of modern networks, SBN provides better OOD detection and calibration with minimal computational overhead compared to traditional Bayesian methods. Future work may explore more complex distribution families to further refine these uncertainty estimates.

Summary updated

Abstract

In this work, we investigate Batch Normalization technique and propose its probabilistic interpretation. We propose a probabilistic model and show that Batch Normalization maximazes the lower bound of its marginalized log-likelihood. Then, according to the new probabilistic model, we design an algorithm which acts consistently during train and test. However, inference becomes computationally inefficient. To reduce memory and computational cost, we propose Stochastic Batch Normalization -- an efficient approximation of proper inference procedure. This method provides us with a scalable uncertainty estimation technique. We demonstrate the performance of Stochastic Batch Normalization on popular architectures (including deep convolutional architectures: VGG-like and ResNets) for MNIST and CIFAR-10 datasets.
BibTeX
@inproceedings{atanov2019uncertainty,
  author = {Andrei Atanov and Arsenii Ashukha and Dmitry Molchanov and Kirill Neklyudov and Dmitry P. Vetrov},
  title = {{Uncertainty Estimation via Stochastic Batch Normalization}},
  booktitle = {ISNN},
  year = {2019},
  doi = {10.1007/978-3-030-22796-8\_28},
}
Figure