Abstract
Deep Neural Networks (DNNs) are notoriously overconfident, often assigning high probabilities to incorrect predictions, particularly when encountering out-of-distribution (OOD) data. While techniques like Bayesian Neural Networks, Dropout, and Deep Ensembles attempt to address this, they frequently suffer from computational intractability, performance degradation in modern architectures like ResNets, or high memory overhead. We introduce Stochastic Batch Normalization (SBN), a method that leverages the inherent stochasticity of Batch Normalization (BN) statistics. By treating BN means and variances as random variables during inference, SBN provides an efficient and scalable uncertainty estimation framework. Our evaluation across multiple architectures shows that SBN improves OOD detection and model calibration while preserving predictive accuracy.
1. Introduction
In safety-critical fields such as medical diagnosis or autonomous driving, knowing when a model is uncertain is as vital as the prediction itself. Standard Batch Normalization is a staple of modern DNNs, but it is "inconsistent": it is stochastic during training (depending on mini-batch samples) but deterministic at test time (using fixed running averages). SBN restores this consistency by modeling the distribution of these statistics, allowing for Monte Carlo uncertainty estimation without requiring architectural changes or training multiple models.
2. Methodology
SBN is based on a probabilistic interpretation of Batch Normalization, where training is viewed as a form of approximate Bayesian inference.
2.1 Parametric Approximation
To make inference efficient, the authors model the distributions of the batch statistics. Empirical analysis suggests that the batch mean ($\mu$) follows a Gaussian distribution, while the batch variance ($\sigma^2$) follows a Log-Normal distribution.
2.2 Inference Procedure
During training, SBN uses exponential smoothing to estimate the parameters of these distributions (e.g., the variance of the means). At test time, SBN performs multiple forward passes, sampling new $\mu$ and $\sigma^2$ values for each layer from these learned distributions. This allows for "free" uncertainty estimation from pre-trained BN networks.
3. Experimental Results
The researchers evaluated SBN using standard benchmarks, including MNIST (OOD: notMNIST) and CIFAR-10 (OOD: CIFAR5), across LeNet-5, VGG-11, and ResNet-18 architectures.
3.1 Out-of-Distribution Detection
SBN consistently produces higher entropy for OOD images compared to standard BN. Furthermore, SBN exhibits strong synergy with existing methods. As shown in the entropy Cumulative Distribution Function (CDF) plots, adding SBN to Dropout or Deep Ensembles (dashed lines) shifts the distribution toward higher entropy, indicating improved uncertainty quantification.
3.2 Calibration and Accuracy
SBN significantly improves model calibration. For example, on ResNet-18 (CIFAR5), SBN improved the Negative Log-Likelihood (NLL) of a Deep Ensemble from 0.138 to 0.110. Crucially, SBN maintains high predictive accuracy; in some cases, such as LeNet-5, the test error even decreased from 0.51% (with Dropout) to 0.49% (with SBN).
4. Conclusion
Stochastic Batch Normalization offers a scalable, ResNet-friendly alternative for uncertainty estimation. By leveraging the stochasticity already present in the BN layers of modern networks, SBN provides better OOD detection and calibration with minimal computational overhead compared to traditional Bayesian methods. Future work may explore more complex distribution families to further refine these uncertainty estimates.
Summary updated