Abstract
Uncertainty estimation in deep models is essential in many real-world applications and has benefited from developments over the last several years. Recent evidence suggests that existing solutions for training Bayesian Deep Neural Networks (BNNs) are limited by the computational overhead of Monte Carlo (MC) sampling. This paper introduces a novel graph reparameterization technique that makes the computation graph size independent of the number of MC samples, allowing researchers to run 1000+ iterations on a single GPU with up to 14x speedups and significantly better uncertainty estimates.
Introduction
Bayesian Deep Learning is considered the "gold standard" for uncertainty estimation in high-stakes domains like medicine and autonomous driving. Most practical implementations rely on Variational Inference (VI) to approximate the posterior distribution by minimizing the Evidence Lower Bound (ELBO). A critical component of this loss is the Kullback-Leibler (KL) divergence. While closed-form solutions exist for simple Gaussian priors, more complex and expressive distributions require Monte Carlo (MC) sampling to estimate the KL term.
However, standard deep learning frameworks (e.g., PyTorch, TensorFlow) face a significant bottleneck: the "memory wall." In these libraries, every MC sample adds nodes to the computation graph used for backpropagation. Consequently, using 100 samples results in a graph 100 times larger, quickly exhausting GPU memory. This constraint often forces practitioners to use only 1 to 10 samples, leading to high-variance gradients and unstable training.
Methods: Graph Reparameterization
The core contribution of this work is a Graph Reparameterization framework that decouples the size of the computation graph from the number of MC samples. The authors identify specific probability families where the summation of samples can be restructured.
The method defines a reparameterization tuple $(G, n, t)$, showing that for many distributions, a function $F$ of MC samples can be expressed as a product between a vector of parameters $n(\theta)$ and a vector of auxiliary random variables $t(\xi)$. By performing the summation inside the $t(\xi)$ term—which does not require gradients—the portion of the graph stored for backpropagation remains small and constant.
This approach is theoretically grounded in the Pitman-Koopman-Darmois theorem, making it applicable to broad classes of distributions, including the Exponential Family (e.g., Gamma, Dirichlet) and Location-Scale families (e.g., Normal, Laplace). For functions like $\log w$ that do not naturally fit this structure, the authors employ Taylor expansions to create polynomial approximations that can be efficiently reparameterized.
Experimental Evaluation
The authors evaluated the proposed method across various architectures (ResNet, DenseNet, VGG) on the CIFAR-10 dataset and a medical imaging task using the ADNI (Alzheimer’s Disease Neuroimaging Initiative) dataset. The ADNI task involved classifying 3D MRIs using a modified 3D U-Net encoder to distinguish cognitively normal patients from those with Alzheimer’s disease.
Key Results
The proposed method demonstrates substantial improvements in both computational efficiency and model performance:
- Massive Speedups: The method achieves up to 14x speedups compared to standard gradient accumulation (GA) methods.
- Memory Efficiency: For a DenseNet-121, it utilizes less than 25% of the GPU memory required by direct MC methods for the same number of samples.
- Scalability: While standard methods struggle with more than 5 MC iterations for large models like VGG-16 on an 11GB GPU, this approach scales to 1000+ iterations with minimal memory overhead.
- Improved Uncertainty: By enabling high-sample counts (1000+), the models produce more stable and reliable uncertainty estimates. On the ADNI dataset, increasing MC iterations from 1 to 100 improved accuracy in the "high confidence" bracket by 26.7%.
Discussion and Limitations
While the graph reparameterization trick is highly effective for the KL divergence term, applying it to the data-dependent likelihood term remains challenging as it involves the entire neural network. The authors suggest partial solutions, such as restricting reparameterization to the final layer. Additionally, for certain distributions, the accuracy of the method depends on the degree of the Taylor expansion; a very high-degree expansion could eventually increase graph complexity. Despite these limitations, this work provides a scalable path forward for training high-fidelity Bayesian deep models.
Summary updated