Inverse-Reference Priors for Fisher Regularization of Bayesian Neural Networks

Keunseo Kim, Eun-Yeol Ma, Jeongman Choi, Heeyoung KimAAAI 2023 · DOI

TL;DR

This paper introduces a novel "inverse-reference" (IR) prior for Bayesian Neural Networks (BNNs) that effectively regularizes the Fisher Information Matrix (FIM). This approach significantly improves generalization performance and robustness to hyperparameter misspecification, outperforming existing methods on benchmark image datasets.

What Problem Does This Paper Address?

Deep Neural Networks (DNNs) often struggle with generalization, meaning they perform poorly on unseen data. Recent research suggests this is linked to the "local curvature" of the loss function, specifically the trace of the Fisher Information Matrix (FIM), which can become excessively high during early training. While methods like the "Fisher penalty" have been proposed to regularize the FIM in standard DNNs, applying this directly to Bayesian Neural Networks (BNNs) is challenging due to the probabilistic nature of their weights. BNNs are valuable for their ability to quantify uncertainty, but their generalization can also degrade, especially with large datasets and poorly chosen priors. This paper aims to bridge this gap by proposing a new prior for BNNs that achieves FIM regularization, thereby enhancing generalization and robustness.

Methods

The core contribution of this paper is the introduction of the inverse-reference (IR) prior. The IR prior is derived by manipulating the concept of a "reference prior," which is a noninformative prior that maximizes the mutual information between parameters and data. The authors propose that by taking the inverse of this reference prior's behavior, they can achieve Fisher regularization.

Specifically, the IR prior is formulated to minimize the KL divergence between the prior and the posterior distribution. Since the true posterior is intractable, they use a variational distribution as an approximation. This leads to an optimization objective where the IR prior is effectively set to the learned variational posterior. The paper demonstrates that this approach, when integrated into the BNN training process via the Evidence Lower Bound (ELBO), naturally regularizes the FIM. This is achieved by showing that the ELBO can be expressed as the expectation of the log-likelihood regularized by the FIM of the variational distribution.

Datasets

The paper evaluates its proposed method on three benchmark image datasets:

  • CIFAR-10
  • CIFAR-100
  • SVHN

These datasets were used with various neural network architectures to test the IR prior's performance across different model scales.

Key Insights and Ideas

  • FIM Regularization for BNNs is Crucial: The paper reinforces the idea that the Fisher Information Matrix (FIM) plays a critical role in the generalization ability of neural networks, extending this insight to BNNs.
  • Inverse-Reference Prior as a Novel Regularizer: The introduction of the IR prior offers a principled way to regularize the FIM within the BNN framework, overcoming the challenges of direct FIM calculation.
  • Prior Choice Matters for Robustness: The IR prior demonstrates remarkable robustness to hyperparameter misspecification, particularly with learning rates, a significant advantage over other common priors.
  • Bridging Generalization and Uncertainty: The IR prior not only improves predictive performance but also provides comparable or even better uncertainty quantification than standard Gaussian priors.
  • Adversarial Robustness Link: The paper suggests a theoretical link between FIM regularization and adversarial robustness, implying that the IR prior might also enhance a model's resilience to adversarial attacks.

Key Results

  • Superior Validation Accuracy: The IR prior consistently achieved the highest validation accuracy across all tested datasets (CIFAR-10, CIFAR-100, SVHN) and neural network architectures (DenseNet121, ResNet18, VGG16), outperforming horseshoe, self-stabilizing, and Gaussian priors. For instance, on CIFAR-10, the IR prior achieved accuracies of 85.5%, 84.6%, and 84.1% for DenseNet121, ResNet18, and VGG16, respectively.
  • Stable Performance: The IR prior exhibited the most stable validation accuracy, with lower standard errors compared to other priors, indicating its reliability.
  • Robustness to Learning Rate: Even with extremely low learning rates (e.g., 0.00001), the BNN with the IR prior maintained superior validation accuracy, demonstrating its resilience to hyperparameter misspecification.
  • Reduced FIM Trace: Experiments showed that the IR prior effectively suppressed the increase in the trace of the FIM during training, especially when using small learning rates, which is a key mechanism for its regularization effect.
  • Competitive Uncertainty Quantification: The IR prior provided uncertainty quantification comparable to, and in some cases slightly better than, the Gaussian prior.
  • Improved Adversarial Robustness: Preliminary results suggest that the IR prior also enhances robustness against adversarial attacks, with higher validation accuracies on perturbed datasets.

Limitations and Future Work

  • Theoretical Approximation Error: The authors acknowledge that their IR prior is an approximation of the true reference prior. Future work will involve theoretically studying the approximation error arising from the difference between the true and variational posterior distributions.
  • Further Adversarial Robustness Analysis: While the paper presents initial findings on adversarial robustness, more in-depth theoretical and empirical investigations are warranted.
  • Broader Applicability: Exploring the IR prior's effectiveness on different types of neural network architectures and a wider range of tasks beyond image classification would be a valuable next step.

Summary updated

BibTeX
@inproceedings{kim2023inverse,
  author = {Keunseo Kim and Eun-Yeol Ma and Jeongman Choi and Heeyoung Kim},
  title = {{Inverse-Reference Priors for Fisher Regularization of Bayesian Neural Networks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i7.25997},
}
Figure