Abstract
Bayesian neural networks (BNNs) have been widely used in many applications because of their distinctive probabilistic representation framework. Even though BNNs have been found more robust to adversarial attacks compared with vanilla neural networks, their ability to deal with adversarial noises in practice is still limited. In this paper, we propose Spectral Expectation Bound Regularization (SEBR) to enhance the robustness of Bayesian neural networks. Our theoretical analysis reveals that training with SEBR improves robustness to adversarial noises. We also prove that training with SEBR can reduce the epistemic uncertainty of the model, making the model more confident in its predictions. Experiments on multiple BNN structures and different adversarial attacks validate the correctness of the theoretical findings and the effectiveness of the proposed approach.
Introduction
Deep neural networks are notoriously fragile; tiny, invisible perturbations, known as adversarial attacks, can flip a model's prediction entirely. While Bayesian Neural Networks (BNNs)—which treat weights as probability distributions rather than fixed points—offer better uncertainty estimation and some inherent robustness, they remain vulnerable in practical deployments. A common defense in deterministic networks is to constrain the Lipschitz constant, which limits how much the output can change for a given change in input. However, applying this to BNNs is computationally challenging because the weights are random variables, making the calculation of the "expected" sensitivity slow and expensive.
The SEBR Framework
The core contribution of this work is Spectral Expectation Bound Regularization (SEBR). The authors establish that the sensitivity of a BNN layer is bounded by the expectation of the spectral norm of its weight matrix. To avoid the computational burden of Monte Carlo sampling and iterative algorithms (like Power Iteration) during training, the authors derive an analytical upper bound for this expectation (Theorem 2).
This bound allows the model to estimate the expected spectral norm using simple statistics of the weight distributions, such as means and variances. By adding this bound as a regularization term to the standard Evidence Lower Bound (ELBO) objective, the training process naturally pushes the model toward a lower Lipschitz constant. This approach reduces computational complexity from $O(KN)$ to $O(K+N)$, where $K$ is the number of Monte Carlo samples and $N$ is the number of power iterations, making the regularization significantly faster.
Empirical Robustness and Accuracy
The effectiveness of SEBR was evaluated across standard image classification benchmarks, including MNIST, Fashion-MNIST, and CIFAR-10/100, using architectures like Bayesian MLPs and VGG. The results demonstrate a significant improvement in adversarial accuracy. For instance, on the MNIST dataset under a medium-strength FGSM attack, a standard Bayesian MLP achieved only 8.97% accuracy, whereas the SEBR-enhanced model reached 43.69%.
As shown in the figure above, increasing the regularization strength ($\lambda$) significantly improves the model's accuracy under varying noise ratios. However, a trade-off exists: as $\lambda$ increases to maximize robustness, there is a slight penalty in "clean accuracy" (accuracy on non-perturbed images), typically ranging from 0.2% to 5% depending on the dataset.
Impact on Model Uncertainty
A key theoretical finding (Theorem 3) is that training with SEBR naturally reduces epistemic uncertainty—the uncertainty stemming from a lack of data or knowledge. By making the model less sensitive to input noise, it becomes more confident in its internal representations.
Visualizations of uncertainty distributions confirm that models trained with SEBR have much tighter, lower uncertainty distributions on adversarial samples compared to standard BNNs. This verifies that the model is less "confused" by adversarial noise, providing a secondary confirmation of the model's enhanced robustness.
Computational Efficiency and Conclusion
Beyond robustness, SEBR offers substantial practical advantages. In experimental trials, SEBR was approximately 4x faster than direct spectral norm regularization. For example, an epoch that required ~1654 seconds with direct regularization was completed in only ~410 seconds with SEBR.
In conclusion, SEBR provides a computationally efficient and theoretically grounded method for regularizing BNNs. By constraining the Lipschitz constant through an analytical bound, it improves adversarial robustness, reduces epistemic uncertainty, and maintains practical training speeds, offering a fundamental layer of protection for probabilistic deep learning models.
Summary updated