Scale Mixtures of Neural Network Gaussian Processes

Hyungi Lee, Eunggu Yun, Hongseok Yang, Juho Lee

By now, it is a well-established result in deep learning theory that infinitely wide neural networks behave like Gaussian Processes (GPs). This equivalence, known as the Neural Network Gaussian Process (NNGP), has provided a powerful lens for understanding how neural networks generalize. However, GPs have a significant limitation: they are "thin-tailed." This means they can be overly sensitive to outliers and overconfident when encountering data far from the training distribution.

In their ICLR 2022 paper, "Scale Mixtures of Neural Network Gaussian Processes," Hyungi Lee and colleagues propose a remarkably simple fix. By treating the variance of the network's last layer as a random variable rather than a fixed constant, they transform the standard NNGP into a much richer class of stochastic processes, including the heavy-tailed Student’s $t$ process.

TL;DR

The authors generalize the "infinite width" limit of neural networks by introducing a prior distribution on the scale of the last-layer weights. This simple change allows any neural architecture to converge to a "Scale Mixture of NNGPs," such as Student’s $t$ processes, which are significantly more robust to outliers and out-of-distribution data than standard Gaussian Processes.

What Problem Does This Paper Address?

Standard NNGP theory assumes that all weights and biases are initialized from fixed Gaussian distributions. While this makes the math elegant, it restricts the resulting stochastic process to be Gaussian. In the real world, data is often "messy"—it contains outliers, noise that isn't normally distributed, and samples that look nothing like the training set (out-of-distribution, or OOD).

Gaussian Processes struggle here because their probability density drops off exponentially. A single outlier can "pull" the entire mean function toward it, and the model's uncertainty estimates (variance) often fail to capture the true risk of OOD samples. The authors sought a way to broaden the types of processes wide networks can represent without losing the theoretical tractability that makes NNGPs useful.

Methods

The core idea is Scale Mixing. In statistics, a scale mixture of Gaussians is created by taking a Gaussian distribution and putting a prior on its variance (scale) parameter.

  1. The Prior: Instead of setting the last-layer weight variance $\sigma_v^2$ to a constant, the authors define it as a random variable $\sigma_v^2 \sim H$, where $H$ is a distribution on positive real numbers.
  2. Theoretical Framework: Using Greg Yang’s "Tensor Programs" framework, they prove that this modification works for any architecture (MLPs, CNNs, RNNs, etc.). As the width $n \to \infty$, the network converges to a scale mixture of NNGPs.
  3. Student’s $t$ Processes (TPs): They show that if you choose an Inverse Gamma distribution for the scale prior, the infinite-width network becomes a Student’s $t$ process. TPs are famous for their "heavy tails," which allow them to ignore outliers more effectively.
  4. Training Dynamics: The authors don't just look at initialization. They analyze what happens when these networks are trained using gradient descent. They extend the Neural Tangent Kernel (NTK) theory to show that fully trained wide networks with this scale prior also converge to scale mixtures of GPs.
  5. Inference Algorithms: They provide practical ways to use these models:
  6. Exact Inference: Possible for regression when using the Student’s $t$ process.
  7. Importance Sampling: An efficient method for generic scale priors.
  8. SVTP: A "Sparse Variational Student’s $t$ Process" algorithm for large-scale classification tasks.

Datasets

The authors validated their theories and models across several benchmarks:

  • Regression: Eight standard UCI datasets (e.g., Boston Housing, Concrete Strength, Wine Quality).
  • Image Classification: MNIST, Fashion-MNIST, KMNIST, EMNIST, CIFAR-10, and SVHN.
  • Robustness Tests: They created "corrupted" versions of these datasets (adding shot noise, impulse noise, etc.) and OOD variants (removing specific classes during training) to test how well the models handle distributional shifts.

Key Insights and Ideas

  • The "Conditioning Trick": The mathematical beauty of this paper lies in the fact that if you condition on the scale variable $\sigma_v^2$, the network becomes a standard NNGP. This allowed the authors to "recycle" existing NNGP and NTK proofs and then marginalize over the scale to get their new results.
  • Last Layer is Enough: One might think you need to change the priors for the entire network to get heavy tails. The authors show that simply changing the readout (last) layer is sufficient to transform the entire stochastic process.
  • Theoretical-Empirical Match: The theory holds up remarkably well in practice. As shown in Figure 1, the distribution of outputs from a wide finite-width network (sampled) perfectly matches the theoretical prediction (predicted).
Comparison between NNGP (Gaussian) and Scale Mixture (Student's t) posterior predictives, and validation of the theoretical limit.

Key Results

  • Superior Calibration: On UCI regression tasks, the scale mixture models (specifically Student’s $t$) generally achieved better Negative Log-Likelihood (NLL) than standard NNGPs. This indicates that their uncertainty estimates are more "honest."
  • OOD Robustness: In image classification, the Sparse Variational Student’s $t$ Process (SVTP) significantly outperformed the Gaussian version (SVGP) on out-of-distribution data. While both performed similarly on "clean" in-distribution data, the SVTP was much less overconfident on unseen classes.
  • Finite Ensemble Gains: When testing ensembles of finite-width CNNs, those initialized with the scale mixture prior were much more robust to data corruptions (like blur or noise) compared to standard ensembles.

Limitations and Future Work

  • Computational Overhead: While the authors optimized their algorithms (e.g., O(1) likelihood computation per sample in importance sampling), scale mixtures are still computationally more demanding than a single GP.
  • Hyperparameter Sensitivity: The behavior of the heavy tails depends on the parameters of the scale prior (like the $a$ and $b$ parameters of the Inverse Gamma). Choosing these optimally for different tasks remains an open area.
  • Beyond Student's $t$: While the paper focuses heavily on Student's $t$ processes, the framework allows for many other types of processes. Exploring these other "flavors" of stochastic processes could yield even more specialized tools for robust AI.

Summary updated

Abstract

Recent works have revealed that infinitely-wide feed-forward or recurrent neural networks of any architecture correspond to Gaussian processes referred to as Neural Network Gaussian Processes (NNGPs). While these works have extended the class of neural networks converging to Gaussian processes significantly, however, there has been little focus on broadening the class of stochastic processes that such neural networks converge to. In this work, inspired by the scale mixture of Gaussian random variables, we propose the scale mixture of NNGPs for which we introduce a prior distribution on the scale of the last-layer parameters. We show that simply introducing a scale prior on the last-layer parameters can turn infinitely-wide neural networks of any architecture into a richer class of stochastic processes. With certain scale priors, we obtain heavy-tailed stochastic processes, and in the case of inverse gamma priors, we recover Student's $t$ processes. We further analyze the distributions of the neural networks initialized with our prior setting and trained with gradient descents and obtain similar results as for NNGPs. We present a practical posterior-inference algorithm for the scale mixture of NNGPs and empirically demonstrate its usefulness on regression and classification tasks. In particular, we show that in both tasks, the heavy-tailed stochastic processes obtained from our framework are robust to out-of-distribution data.
BibTeX
@inproceedings{lee2022scale,
  author = {Hyungi Lee and Eunggu Yun and Hongseok Yang and Juho Lee},
  title = {{Scale Mixtures of Neural Network Gaussian Processes}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=YVPBh4k78iZ},
}
Figure