Uncertainty Quantification in CNN Through the Bootstrap of Convex Neural Networks

Hongfei Du, Emre Barut, Fang JinAAAI 2021 · DOI

Introduction

While Convolutional Neural Networks (CNNs) have revolutionized pattern recognition, they often suffer from "overconfidence," providing high-probability predictions even when incorrect. In high-stakes domains like medical diagnostics or autonomous navigation, quantifying a model's uncertainty is as critical as the prediction itself. This paper addresses the challenges of Uncertainty Quantification (UQ) in deep learning by introducing a theoretically grounded framework that combines the bootstrap method with Convexified Convolutional Neural Networks (CCNN).

Existing UQ methods typically struggle with three issues: theoretical inconsistency due to the non-convex nature of standard CNNs, the high computational cost of training ensembles, and the tendency of deep models to overfit, which leads to underestimated uncertainty.

Methodology

The proposed framework bridges the gap between deep learning and classical statistics through three primary innovations:

  • Convexified CNNs (CCNN): By using a convex relaxation of a two-hidden-layer CNN, the authors ensure that any local optimum reached during training is a global optimum. This convexity provides the mathematical foundation necessary for bootstrap estimates to be asymptotically consistent.
  • Warm-Start Bootstrap: Traditional bootstrapping requires retraining models from scratch for every resampled dataset. Leveraging the convex landscape, the authors employ a "warm-start" strategy—initializing each bootstrap iteration with weights from the previous one—reducing computational overhead by an order of magnitude.
  • Transfer Learning Bridge: To overcome the structural limitations of shallow CCNNs, the framework uses deep, pre-trained models (e.g., VGG16) as feature extractors. The CCNN is then trained on these high-level features. To maintain statistical independence, the authors utilize strategies like "Train and Forget" or "Train and Perturb" to ensure pre-trained features do not leak training set specifics.

Experimental Evaluation

The framework was evaluated across five diverse datasets: MNIST, Noisy MNIST (with motion blur), Fashion-MNIST, CIFAR10, and a Cats and Dogs dataset. The CCNN Bootstrap was benchmarked against standard CNNs and Deep Ensembles, the current industry standard for UQ.

Key Insights and Results

The most significant finding is that convexity enables reliable, stable uncertainty measurements. Unlike non-convex models where warm-starts might lead to poor local optima, in CCNNs, they serve as a direct shortcut to the global peak, making UQ computationally feasible for large-scale applications.

Uncertainty quantification for MNIST digits 0-4. The plots demonstrate how the model provides wider prediction intervals and identifies secondary candidates for ambiguous samples, such as the digit '4' which shows probability mass for '9'.

Key performance metrics include:

  • Calibration: The CCNN approach achieved higher Average Log-Likelihood across most datasets, indicating more accurate probability estimates.
  • Precision: The model produced tighter 95% confidence intervals than ensembles while maintaining superior coverage, demonstrating a balance of decisiveness and caution.
  • Diagnostic Utility: As shown in the figure above, the model acts as a diagnostic tool, producing wider intervals for ambiguous samples (e.g., a '4' that resembles a '9'), offering transparency into the model's internal uncertainty.

Discussion and Future Work

While the CCNN Bootstrap offers a robust theoretical framework, it currently relies on a two-layer approximation and kernel tricks for non-linearities, which can be memory-intensive for very large datasets. Future research directions include integrating this framework into deep reinforcement learning to enable safer exploration by quantifying the upper bounds on potential rewards.

Summary updated

BibTeX
@inproceedings{du2021uncertainty,
  author = {Hongfei Du and Emre Barut and Fang Jin},
  title = {{Uncertainty Quantification in CNN Through the Bootstrap of Convex Neural Networks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i13.17434},
}
Figure