Scale Mixtures of Neural Network Gaussian Processes

Hyungi Lee, Eunggu Yun, Hongseok Yang, Juho Lee

Abstract

Recent theoretical work has established an equivalence between infinitely wide neural networks and Gaussian Processes (NNGPs). However, the thin-tailed nature of Gaussian distributions often leads to overconfident predictions and poor calibration in the presence of outliers or out-of-distribution (OOD) data. This paper introduces Scale Mixtures of NNGPs, a generalization achieved by placing a prior distribution on the scale of the last-layer parameters. We prove that wide networks with such priors converge to heavy-tailed stochastic processes, such as Student’s $t$ processes. Our framework demonstrates significantly improved robustness and uncertainty calibration across various regression and classification benchmarks while maintaining computational efficiency.

Introduction: Beyond the Gaussian Limit

The Neural Network Gaussian Process (NNGP) framework provides a rigorous Bayesian interpretation of deep learning by showing that wide networks at initialization behave like Gaussian Processes. Despite its mathematical elegance, the NNGP's reliance on Gaussianity is a practical drawback. Real-world datasets frequently contain noise, outliers, and OOD samples that Gaussian models struggle to handle gracefully, often resulting in overconfident misclassifications.

This work addresses this limitation by broadening the class of stochastic processes that wide networks can represent. By simply modifying the prior of the last layer, we can transform the network's output from a standard GP into a more flexible scale mixture of GPs.

Methodology

The core innovation lies in treating the last-layer variance as a random variable rather than a fixed constant.

  1. Scale Mixtures and Student’s $t$ Processes: By placing a prior distribution $H$ on the last-layer weight variance ($\sigma_v^2$), the network converges to a scale mixture of NNGPs. Specifically, using an Inverse Gamma prior results in a Student’s $t$ process, which possesses the heavy tails necessary for robust inference.
  2. Theoretical Convergence: Utilizing the Tensor Programs framework, we prove this convergence holds as width $n \to \infty$. Crucially, this correspondence persists during training under the Neural Tangent Kernel (NTK) regime.
  3. Efficient Inference: To ensure practical utility, we developed optimized algorithms. For regression, an importance sampling approach reduces complexity to $O(K + N^3)$, where $K$ is the number of samples. For classification, we introduce Sparse Variational Student’s $t$ Processes (SVTP) to manage non-Gaussian likelihoods efficiently.

Key Insights and Empirical Validation

A primary finding is the "Last Layer Lever": modifying only the readout layer's prior is sufficient to change the entire network's stochastic behavior. This allows for heavy-tailed robustness without complex architectural changes.

Empirical validation of Scale Mixtures of NNGPs. The first two panels compare Gaussian vs. Student’s t process fits to a sine wave. The remaining panels show the match between theoretical predictions and sampled finite-width networks at various stages of training.

As shown in the figure above, the Student’s $t$ process (second panel) provides a more conservative uncertainty estimate compared to the standard Gaussian fit (first panel). Furthermore, the histograms demonstrate an excellent match between our infinite-width theory ("Predicted") and empirical samples from finite-width networks ("Sampled"), both at initialization and after training.

Key Results

The proposed framework was evaluated on UCI regression datasets and various image classification tasks (MNIST, CIFAR-10, etc.), including corrupted and OOD scenarios.

  • Superior Calibration: In OOD classification, the SVTP model consistently achieved lower Negative Log-Likelihood (NLL) than standard Sparse Variational GPs, indicating better-calibrated uncertainty.
  • Robustness to Noise: On regression tasks with heavy-tailed noise, the Student’s $t$ process significantly outperformed standard NNGPs.
  • Performance Preservation: The use of scale mixtures did not degrade performance on clean, well-behaved data, acting as a "safety net" for anomalous inputs.

Conclusion

Scale Mixtures of NNGPs offer a simple yet powerful extension to the deep learning theory toolkit. By incorporating heavy-tailed priors, we can build wide neural networks that are inherently more robust to the "messiness" of real-world data. Future research will focus on scaling variational methods beyond current memory constraints and automating hyperparameter selection for the scale priors.

Summary updated

BibTeX
@inproceedings{lee2022scale,
  author = {Hyungi Lee and Eunggu Yun and Hongseok Yang and Juho Lee},
  title = {{Scale Mixtures of Neural Network Gaussian Processes}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=YVPBh4k78iZ},
}
Figure