PAC Bayesian Performance Guarantees for Deep (Stochastic) Networks in Medical Imaging

Anthony Sicilia, Xingchen Zhao, Anastasia Sosnovskikh, Seong Jae HwangarXiv 2021 · arXiv

Introduction

Deep learning models in medical imaging are often criticized for overfitting on small datasets. While practitioners typically rely on holdout sets to estimate real-world performance, these methods do not provide mathematical proof of generalization and often waste precious data. This research applies the PAC-Bayesian framework to provide "non-vacuous" (meaningful) performance guarantees for large-scale networks like U-Net and ResNet-18, linking generalization to the "flatness" of the loss landscape.

The PAC-Bayesian Framework

Unlike standard deterministic networks with fixed weights, this approach utilizes stochastic networks where weights are sampled from a posterior distribution ($Q$). The goal is to find a bound on the expected error on unseen data that is both mathematically rigorous and practically useful.

Historically, theoretical generalization bounds (such as those based on VC Dimension) have been "vacuous" for deep learning, predicting error rates exceeding 100%. By using the PAC-Bayesian framework, the authors generate meaningful percentages for complex tasks like lesion segmentation.

Methods and Data

The study focuses on the ISIC 2018 Challenge dataset, covering skin lesion segmentation (~2,300 images using U-Net) and classification (~9,000 images using ResNet-18). To achieve tight, non-vacuous bounds, the authors introduced several technical innovations:

  • Data-Dependent Priors: The training data is split; one half informs a prior distribution ($P$), while the other half trains the final posterior ($Q$) and calculates the bound.
  • Batch Normalization Handling: The authors treat BN parameters as "point mass distributions," allowing these standard layers to be included in the PAC-Bayesian math without breaking the sampled-weight paradigm.
  • PAC-Bayes with Backprop (PBB): The networks are optimized specifically to minimize the PAC-Bayes bound itself, training the model to be "guaranteed" to perform well.

Key Insights: The "Flat Minima" Prescription

A central hypothesis in deep learning generalization is that models performing well on unseen data often land in "flat minima"—regions where small weight perturbations do not significantly alter the output. The PAC-Bayesian framework mathematically rewards this flatness. By modulating the variance of the prior (the "sigma prior"), the authors probed the loss landscape. For U-Net, performance remained stable until a specific threshold, suggesting the model resides in a wide, robust valley.

Key Results

The authors successfully produced meaningful bounds for a full-scale U-Net on a relatively small dataset of 2,300 samples. As shown in the results, these PAC-Bayesian bounds are comparable to the standard Hoeffding Bound (calculated via holdout sets), but offer additional qualitative insights into model stability.

Comparison of empirical performance (DSC/ACC) vs. PAC-Bayesian lower bounds across different architectures and sensitivity to prior variance.

The analysis revealed that segmentation (U-Net) and classification (ResNet-18) exhibit different generalization behaviors. While U-Net showed a clear "flatness" threshold, ResNet-18 demonstrated more consistent degradation as variance increased.

Conclusion and Future Work

This work demonstrates that PAC-Bayesian bounds can serve as a "self-bounding learner," where the training data itself provides a certificate of performance. This could alleviate the need for large independent validation sets in resource-constrained medical settings. Future research will focus on reducing the "tightness gap" between guaranteed and observed error and testing these frameworks in ultra-low resource environments with fewer than 100 cases.

Summary updated

Abstract

Application of deep neural networks to medical imaging tasks has in some sense become commonplace. Still, a "thorn in the side" of the deep learning movement is the argument that deep networks are prone to overfitting and are thus unable to generalize well when datasets are small (as is common in medical imaging tasks). One way to bolster confidence is to provide mathematical guarantees, or bounds, on network performance after training which explicitly quantify the possibility of overfitting. In this work, we explore recent advances using the PAC-Bayesian framework to provide bounds on generalization error for large (stochastic) networks. While previous efforts focus on classification in larger natural image datasets (e.g., MNIST and CIFAR-10), we apply these techniques to both classification and segmentation in a smaller medical imagining dataset: the ISIC 2018 challenge set. We observe the resultant bounds are competitive compared to a simpler baseline, while also being more explainable and alleviating the need for holdout sets.
BibTeX
@misc{sicilia2021pac,
  author = {Anthony Sicilia and Xingchen Zhao and Anastasia Sosnovskikh and Seong Jae Hwang},
  title = {{PAC Bayesian Performance Guarantees for Deep (Stochastic) Networks in Medical Imaging}},
  year = {2021},
  eprint = {2104.05600},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2104.05600},
}
Figure