Spatial-Temporal-Fusion BNN: Variational Bayesian Feature Layer

Shiye Lei, Zhuozhuo Tu, Leszek Rutkowski, Feng Zhou, Li Shen, Fengxiang He, Dacheng TaoarXiv 2021 · OpenReview

Abstract

Bayesian neural networks (BNNs) have become a principal approach to alleviate overconfident predictions in deep learning, but they often suffer from scaling issues due to a large number of distribution parameters. In this paper, we discover that the first layer of a deep network possesses multiple disparate optima when solely retrained. This indicates a large posterior variance when the first layer is altered by a Bayesian layer, which motivates us to design a spatial-temporal-fusion BNN (STF-BNN) for efficiently scaling BNNs to large models: (1) first normally train a neural network from scratch to realize fast training; and (2) the first layer is converted to Bayesian and inferred by employing stochastic variational inference, while other layers are fixed. Compared to vanilla BNNs, our approach can greatly reduce the training time and the number of parameters, which contributes to scale BNNs efficiently. We further provide theoretical guarantees on the generalizability and the capability of mitigating overconfidence of STF-BNN. Comprehensive experiments demonstrate that STF-BNN (1) achieves the state-of-the-art performance on prediction and uncertainty quantification; (2) significantly improves adversarial robustness and privacy preservation; and (3) considerably reduces training time and memory costs.

Introduction

Deep neural networks are notoriously "overconfident"—they often provide high-certainty predictions even when they are wrong, which is a major safety concern for applications like autonomous driving or medical diagnosis. Bayesian Neural Networks (BNNs) solve this by treating weights as probability distributions rather than fixed points, allowing the model to quantify its own uncertainty and say "I don't know."

However, BNNs are difficult to scale. By doubling the number of parameters to store both a mean and a variance for every weight, they suffer from massive memory overhead, slow convergence, and training instability. The challenge addressed here is: how do we obtain the reliability of a BNN without the massive computational tax?

Methods: Spatial-Temporal Fusion

The authors introduce Spatial-Temporal-Fusion BNN (STF-BNN), based on a key empirical observation regarding "layer stability."

The Discovery of Layer Instability

By retraining individual layers of a pre-trained network, the researchers found that the first layer is the most "unstable"—it can converge to many different, equally good settings (disparate optima). This suggests that the first layer is naturally the best place to capture model uncertainty.

Layer stability analysis across FashionMNIST and CIFAR-10, showing that the first layer (index 1) has the lowest stability.

Two-Phase Training Pipeline

The STF-BNN approach utilizes a two-phase training strategy, referred to as temporal fusion:

  1. Phase 1 (Deterministic Training): Train a standard, deterministic neural network using normal SGD. This provides a fast and efficient baseline.
  2. Phase 2 (Bayesian Conversion): Freeze all layers except the first one. Convert that first layer into a Bayesian layer (using Variational Inference) and train only its distribution parameters.
Overview of the STF-BNN training pipeline, combining a deterministic backbone with a Bayesian feature layer (labeled as EF-BNN in the diagram).

The resulting hybrid architecture (spatial fusion) consists of a single Bayesian feature layer at the start, followed by a standard deterministic backbone. This approach uses Bayes by Backprop for the first layer while leveraging the pre-trained features of the rest of the network.

Experimental Evaluation

The models were evaluated on standard computer vision benchmarks including FashionMNIST, CIFAR-10, and CIFAR-100. To test out-of-distribution robustness, the authors used CIFAR-10-C and CIFAR-100-C, which contain corrupted images (noise, blur, etc.).

Key Results

  • Efficiency: STF-BNN trained significantly faster than other Bayesian methods. On Wide ResNet-28-10, it achieved superior results in a fraction of the time required by Rank-1 BNNs or Deep Ensembles.
  • Uncertainty Quantification: It achieved lower Expected Calibration Error (ECE) than baselines, meaning its predicted probabilities much more closely matched actual accuracy.
  • Robustness and Privacy: The model showed improved resistance to adversarial attacks (like PGD) and better privacy preservation against Membership Inference Attacks.
  • Performance on Corruptions: On the CIFAR-C datasets, STF-BNN maintained higher accuracy and lower ECE as image corruption intensity increased, outperforming standard deterministic models.

Conclusion and Future Work

STF-BNN demonstrates that "Bayesianizing" only the most unstable layer—the first layer—provides a highly efficient path to reliable uncertainty quantification. While the paper focuses on VGG and Wide ResNet, future work could explore modern Transformer-based architectures or investigate if strategic subsets of layers beyond just the first could offer even better trade-offs. Additionally, addressing hyperparameter sensitivity in the Variational Inference process remains an area for further refinement.

Summary updated

Abstract

Bayesian neural networks (BNNs) have become a principal approach to alleviate overconfident predictions in deep learning, but they often suffer from scaling issues due to a large number of distribution parameters. In this paper, we discover that the first layer of a deep network possesses multiple disparate optima when solely retrained. This indicates a large posterior variance when the first layer is altered by a Bayesian layer, which motivates us to design a spatial-temporal-fusion BNN (STF-BNN) for efficiently scaling BNNs to large models: (1) first normally train a neural network from scratch to realize fast training; and (2) the first layer is converted to Bayesian and inferred by employing stochastic variational inference, while other layers are fixed. Compared to vanilla BNNs, our approach can greatly reduce the training time and the number of parameters, which contributes to scale BNNs efficiently. We further provide theoretical guarantees on the generalizability and the capability of mitigating overconfidence of STF-BNN. Comprehensive experiments demonstrate that STF-BNN (1) achieves the state-of-the-art performance on prediction and uncertainty quantification; (2) significantly improves adversarial robustness and privacy preservation; and (3) considerably reduces training time and memory costs.
BibTeX
@misc{lei2021spatial,
  author = {Shiye Lei and Zhuozhuo Tu and Leszek Rutkowski and Feng Zhou and Li Shen and Fengxiang He and Dacheng Tao},
  title = {{Spatial-Temporal-Fusion BNN: Variational Bayesian Feature Layer}},
  year = {2021},
  eprint = {2112.06281},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2112.06281},
}
Figure