Predictive Complexity Priors

Eric T. Nalisnick, Jonathan Gordon, José Miguel Hernández-Lobato

Abstract

Specifying a Bayesian prior is notoriously difficult for complex models such as neural networks. Reasoning about parameters is made challenging by the high-dimensionality and over-parameterization of the space. Priors that seem benign and uninformative can have unintuitive and detrimental effects on a model’s predictions. For this reason, we propose predictive complexity priors: a functional prior that is defined by comparing the model’s predictions to those of a reference model. Although originally defined on the model outputs, we transfer the prior to the model parameters via a change of variables. The traditional Bayesian workflow can then proceed as usual. We apply our predictive complexity prior to high-dimensional regression, reasoning over neural network depth, and sharing of statistical strength for few-shot learning.

Introduction

In Bayesian modeling, the prior represents our beliefs about the model before observing data. While selecting priors for simple models is straightforward, doing so for Bayesian Neural Networks (BNNs) is notoriously difficult. Weight space in deep learning is high-dimensional, over-parameterized, and largely uninterpretable. Consequently, researchers often rely on "priors of convenience," such as standard normal distributions, which can lead to unintended and detrimental functional behaviors. We possess better intuition regarding how a model should behave—for instance, preferring simplicity unless complexity is warranted—but lack the tools to translate this functional intuition into specific parameter values.

Methods: From Intuition to Parameter Space

We introduce Predictive Complexity Priors (PredCP), a framework that bridges the gap between functional intuition and parameter-space priors. The approach follows a three-step procedural pipeline:

  1. Define a Reference Model: Select a simpler baseline model ($p_0$). For a neural network, this might be a version of the same network with one fewer layer.
  2. Measure Divergence: Quantify the "complexity" or "extra capacity" of the full model relative to the reference using Kullback-Leibler Divergence (KLD).
  3. Reparameterize: Place a prior on this divergence value that favors simplicity. Using the change of variables formula, this functional prior is translated back into a proper prior on the model's hyperparameters ($\tau$).
The PredCP pipeline: (a) defining priors on KLD, (b) resulting ECPs on hyperparameters, and (c-d) induced marginal distributions on parameters.

While the ideal Evidence Complexity Prior (ECP) is computationally demanding due to marginalization requirements, PredCP utilizes Jensen’s Inequality to establish a tractable upper bound on the KLD. This allows the prior to be integrated into standard Bayesian inference workflows, such as Variational Inference or MCMC, using differentiable Monte Carlo approximations.

Key Insights: Function Space Regularization

The core philosophy of PredCP is to regularize the effect of a parameter on the model's output rather than the raw value of the parameter itself. This shift to function space has several profound implications:

  • Automatic Depth Selection: When applied layer-wise, PredCP acts as a functional gate. It penalizes the variance introduced by additional layers. If a new layer does not significantly alter the predictive distribution, the prior effectively "shuts it down," allowing the model to determine its optimal depth automatically.
  • Variance Regularization: In BNNs, PredCP primarily functions by regularizing the variance of hidden units, a mechanism conceptually related to Dropout but grounded in a formal Bayesian framework.
Comparison of function samples from a standard Horseshoe prior versus PredCP with a Log-Cauchy base.

Experimental Results

We evaluated PredCP across three distinct domains to demonstrate its versatility:

  1. High-Dimensional Logistic Regression: On small medical datasets (e.g., allaml, colon) where features far outnumber samples, PredCP outperformed standard half-Cauchy shrinkage priors in 4 out of 6 settings, effectively mitigating overfitting.
  2. Neural Network Regression: On UCI benchmarks, PredCP achieved the lowest RMSE on 3 out of 7 tasks and tied on another, surpassing standard fixed-scale and inverse-gamma priors.
  3. Few-Shot Learning: In the $\sigma$-MAML framework, PredCP improved mean accuracy on Fewshot-CIFAR100 and Mini-ImageNet. For 5-shot Mini-ImageNet, it reached 61.9% accuracy, compared to 58.4% for standard MAML.

Limitations and Future Directions

Despite its advantages, PredCP has limitations. The use of a KLD upper bound for scalability results in the loss of information regarding predictive correlations between data points in a batch. Additionally, the computational cost increases with network depth due to the need for extra forward passes. Future work may explore alternative divergence measures like Hellinger distance or investigate the impact of PredCP on out-of-distribution (OOD) uncertainty estimation.

Summary updated

Abstract

Specifying a Bayesian prior is notoriously difficult for complex models such as neural networks. Reasoning about parameters is made challenging by the high-dimensionality and over-parameterization of the space. Priors that seem benign and uninformative can have unintuitive and detrimental effects on a model’s predictions. For this reason, we propose predictive complexity priors: a functional prior that is defined by comparing the model’s predictions to those of a reference model. Although originally defined on the model outputs, we transfer the prior to the model parameters via a change of variables. The traditional Bayesian workflow can then proceed as usual. We apply our predictive complexity prior to high-dimensional regression, reasoning over neural network depth, and sharing of statistical strength for few-shot learning.
BibTeX
@inproceedings{nalisnick2021predictive,
  author = {Eric T. Nalisnick and Jonathan Gordon and José Miguel Hernández-Lobato},
  title = {{Predictive Complexity Priors}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2021},
  url = {https://openreview.net/forum?id=AhUdN4BNGNq},
}
Figure