Predictive Complexity Priors

Eric T. Nalisnick, Jonathan Gordon, José Miguel Hernández-Lobato

Abstract

Specifying a Bayesian prior is notoriously difficult for complex models such as neural networks. Reasoning about parameters is made challenging by the high-dimensionality and over-parameterization of the space. Priors that seem benign and uninformative can have unintuitive and detrimental effects on a model's predictions. For this reason, we propose predictive complexity priors: a functional prior that is defined by comparing the model's predictions to those of a reference model. Although originally defined on the model outputs, we transfer the prior to the model parameters via a change of variables. The traditional Bayesian workflow can then proceed as usual. We apply our predictive complexity prior to high-dimensional regression, reasoning over neural network depth, and sharing of statistical strength for few-shot learning.

Introduction

In Bayesian modeling, the choice of prior is often the most consequential decision. While intuitions like sparsity or smoothness guide choices for simple models, deep neural networks (DNNs) present a significant challenge. The high-dimensionality and lack of identifiability in parameter space make it nearly impossible to reason about how weight priors—such as the standard Gaussian—influence the resulting function space behavior.

Nalisnick et al. propose Predictive Complexity Priors (PredCP), a framework that shifts the focus from parameter values to functional complexity. By defining priors based on the divergence between a model's predictions and those of a simpler reference model, researchers can specify priors that align with functional intuitions while maintaining a standard Bayesian workflow.

Methodology: From Predictions to Parameters

The construction of a Predictive Complexity Prior follows a principled three-step process:

  1. Reference Model Selection: A simpler "reference" model $p_0$ is chosen. For instance, in a neural network, $p_0$ might be the same architecture but with one fewer layer or reduced capacity.
  2. Complexity Measurement: The Kullback–Leibler Divergence (KLD) is used to quantify the additional complexity introduced by the model's parameters relative to the reference model. This measurement occurs in the output (predictive) space.
  3. Change of Variables: A prior is placed on the KLD value itself (e.g., an Exponential or Log-Cauchy distribution). Using a change of variables, this prior is analytically pulled back onto the model's hyperparameters, such as the layer variance $\tau$.

To scale this to deep learning, the authors introduce an upper bound on the KLD using Jensen’s inequality, allowing for efficient evaluation via mini-batches during training.

The relationship between KLD priors, induced priors on scale parameters (ECPs), and the resulting marginal distributions for model weights.

Key Insights and Applications

PredCP acts as a sophisticated regularizer for predictive variance. By penalizing the divergence from a simpler model, it prevents the network from exhibiting excessive "jitter" or overconfidence in data-sparse regions.

Automatic Depth Determination

When applied layer-wise, PredCP can effectively gate the depth of a network. If a deeper layer does not significantly improve the predictive distribution over its predecessor, the prior pushes the layer's scale parameter toward zero, effectively pruning unnecessary depth.

Data-Dependent Priors

Unlike standard empirical Bayes, PredCP remains a valid Bayesian prior. While it depends on input features $x$, it integrates out the targets $y$, ensuring the prior is informed by the data's structure without "peeking" at the labels.

Comparison of joint density functions and functions sampled from a ResNet under Horseshoe vs. PredCP (Log-Cauchy) priors.

Experimental Results

The authors evaluated PredCP across several domains:

  • High-Dimensional Regression: On medical datasets where features outnumber samples (e.g., leukemia and cancer datasets), PredCP consistently outperformed traditional shrinkage priors like the half-Cauchy in terms of predictive log-likelihood.
  • UCI Regression Benchmarks: In ResNet experiments, PredCP achieved the best RMSE in a majority of datasets, demonstrating that reasoning about depth via complexity is superior to using fixed weight scales.
  • Few-Shot Classification: PredCP provided a consistent accuracy boost over state-of-the-art methods like $\sigma$-MAML on benchmarks such as mini-ImageNet.

Conclusion

Predictive Complexity Priors offer a robust framework for specifying priors in complex models by grounding them in functional behavior. By bridging the gap between parameter space and prediction space, PredCP enables more intuitive and effective regularization for deep learning, particularly in low-data regimes and architecture selection. Future work may explore alternative divergence measures and the application of PredCP to uncertainty estimation and out-of-distribution detection.

Summary updated

Abstract

Specifying a Bayesian prior is notoriously difficult for complex models such as neural networks. Reasoning about parameters is made challenging by the high-dimensionality and over-parameterization of the space. Priors that seem benign and uninformative can have unintuitive and detrimental effects on a model's predictions. For this reason, we propose predictive complexity priors: a functional prior that is defined by comparing the model's predictions to those of a reference model. Although originally defined on the model outputs, we transfer the prior to the model parameters via a change of variables. The traditional Bayesian workflow can then proceed as usual. We apply our predictive complexity prior to high-dimensional regression, reasoning over neural network depth, and sharing of statistical strength for few-shot learning.
BibTeX
@inproceedings{nalisnick2021predictive,
  author = {Eric T. Nalisnick and Jonathan Gordon and José Miguel Hernández-Lobato},
  title = {{Predictive Complexity Priors}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2021},
  url = {https://openreview.net/forum?id=AhUdN4BNGNq},
}

Loading related content...

Figure