Flexible Bayesian Last Layer Models Using Implicitpriors and Diffusion Posterior Sampling

Jian Xu, Zhiqi Lin, Shigui Li, Min Chen, Junmei Yang, Delu Zeng, John Paisley

TL;DR

This paper introduces a novel approach to Bayesian Last Layer (BLL) models by replacing traditional Gaussian priors with flexible implicit priors learned by neural networks. To handle the resulting complexity, they leverage diffusion models for posterior sampling, leading to improved uncertainty quantification and predictive accuracy, especially in challenging scenarios with non-Gaussian data and outliers.

What Problem Does This Paper Address?

Bayesian Last Layer (BLL) models are a powerful tool for quantifying uncertainty in neural networks by focusing on the final layer's weights. However, their reliance on simple Gaussian priors for these weights limits their ability to capture the nuances of complex, real-world data, such as datasets with non-Gaussian distributions or outliers. This limitation can hinder their performance and reliability. The paper aims to overcome this by introducing more expressive priors and a robust inference mechanism to handle them.

Methods

The core innovation lies in two key areas:

  • Implicit Priors: Instead of assuming a fixed distribution (like Gaussian) for the last layer weights, the authors propose using a neural network to generate these priors. This "implicit prior" allows for much greater flexibility, enabling the model to learn complex, non-Gaussian weight distributions that better reflect the data. This is inspired by prior work on implicit processes but adapted for BLL models.
  • Diffusion Posterior Sampling: The flexibility of implicit priors makes the traditional Bayesian inference computationally intractable. To address this, the paper leverages diffusion models, a class of generative models known for their ability to model complex distributions. They employ diffusion models to sample from the posterior distribution of the implicit prior's parameters. This is achieved by formulating a novel variational lower bound for the marginal likelihood, minimizing KL divergence, and utilizing techniques inspired by score matching. Essentially, they use diffusion models to "denoise" and approximate the complex posterior distribution.

Datasets

The proposed method was evaluated on several standard datasets:

  • UCI Regression Datasets: A suite of six standard regression datasets (Boston, Concrete, Energy, Wine, Yacht, Power) were used to evaluate the model's performance in regression tasks.
  • CIFAR-10 and CIFAR-100: These are widely used image classification datasets.
  • SVHN (Street View House Numbers): Used as a "far-out-of-distribution" dataset for evaluating the model's robustness in image classification tasks.

Key Insights and Ideas

  • Flexibility is Key: The paper strongly advocates for moving beyond simple Gaussian priors in BLL models. The experimental results demonstrate that more flexible, learned priors can significantly improve performance.
  • Diffusion Models as Powerful Inference Tools: Diffusion models, typically known for generation, are shown to be effective for complex posterior inference in Bayesian models, offering a way to handle the intractability introduced by flexible priors.
  • Bridging Generative and Discriminative Models: This work highlights how generative modeling techniques (diffusion models) can be directly applied to improve discriminative tasks like classification and regression through enhanced uncertainty quantification.
  • Improved Uncertainty for Challenging Data: The method shows particular promise in scenarios where traditional models struggle, such as datasets with outliers or non-Gaussian characteristics, leading to more reliable uncertainty estimates.

Key Results

The proposed DVI-IBLL (Diffusion Variational Inference - Implicit Prior Bayesian Last Layer) model consistently outperformed or matched existing BLL methods and other Bayesian approaches across various datasets.

  • Regression: DVI-IBLL achieved competitive or superior results in terms of Negative Log Likelihood (NLL) and Root Mean Squared Error (RMSE) on UCI regression datasets, often outperforming strong baselines like VBLL and RBF GP.
  • Image Classification (CIFAR-10/100): The model demonstrated strong performance in accuracy, ECE, and NLL, and notably achieved the highest SVHN AUC and CIFAR-100 AUC among the tested methods, indicating excellent out-of-distribution detection capabilities.
  • Efficiency: While introducing diffusion models, the method maintained computational efficiency, with only a small increase in runtime compared to baseline DNNs.

Limitations and Future Work

  • Computational Cost: While efficient, the use of diffusion models for posterior sampling can still be computationally more intensive than simpler methods, especially for very large models or datasets.
  • Hyperparameter Tuning: The implicit prior network and diffusion model introduce additional hyperparameters that may require careful tuning.
  • Future Work: The authors suggest exploring further applications of this framework to other domains and investigating more advanced diffusion-based inference techniques. They also hint at potential extensions to handle more complex model architectures and tasks.

Summary updated

BibTeX
@misc{xu2025flexible,
  author = {Jian Xu and Zhiqi Lin and Shigui Li and Min Chen and Junmei Yang and Delu Zeng and John Paisley},
  title = {{Flexible Bayesian Last Layer Models Using Implicitpriors and Diffusion Posterior Sampling}},
  year = {2025},
  doi = {10.2139/ssrn.5102545},
  howpublished = {SSRN},
}
Figure