Wide Neural Networks with Bottlenecks are Deep Gaussian Processes

Devanshu Agrawal, Theodore Papamarkou, Jacob Hinkle

Abstract

Recent theoretical advances have shown that Bayesian neural networks (BNNs) converge to Gaussian processes (GPs) in the infinite-width limit. However, these results typically assume all hidden layers are infinitely wide, neglecting architectures with finite-width "bottlenecks" like autoencoders. This paper proves that BNNs with finite bottlenecks converge to a composition of GPs, termed a bottleneck neural network Gaussian process (bottleneck NNGP). We demonstrate that these bottlenecks prevent deep ReLU networks from losing discriminative power, effectively acting as information anchors that allow the network to "remember" input geometry even at infinite depth.

Introduction: The Wide Limit and its Gaps

The "wide limit" has become a cornerstone of deep learning theory. By treating hidden layers as infinitely wide, researchers can map complex BNNs to Neural Network Gaussian Processes (NNGPs), enabling exact kernel-based analysis. Despite its utility, this framework fails to account for practical architectures that intentionally use narrow layers to force data compression or dimensionality reduction. This paper addresses this gap by asking: what happens when some layers remain narrow while others expand to infinity?

Methodology: The Bottleneck NNGP

The authors formally define the bottleneck NNGP and provide a rigorous mathematical proof that a BNN with finite bottlenecks converges in distribution to a composition of NNGPs—a specific class of Deep Gaussian Process (DGP). A key challenge addressed is proving that the "wide limit of a composition" is equivalent to the "composition of the limiting GPs."

Comparison of architectures: (a) MLP, (b) NNGP, (c) Deep NNGP, and (d) the proposed Bottleneck NNGP.

Beyond the convergence proof, the authors analyze the "quadratic correlation"—the correlation between the squares of outputs—induced by the bottleneck. In standard wide networks, outputs are typically independent; however, the bottleneck introduces dependencies that persist through subsequent infinite-width layers.

Theoretical Insights: Information Persistence

A significant finding is the role of bottlenecks as "information anchors." In standard infinite-width ReLU networks, the kernel tends to lose discriminative power as depth increases, eventually treating all inputs as identical. The authors find that a single finite bottleneck prevents this "forgetting."

This persistence of information is uniquely tied to the ReLU activation function. For other common nonlinearities like $tanh$ or $sin$, a bottleneck does not prevent the eventual loss of information in the deep limit. The transition at weight variance $v_w = 1$ acts as a phase transition; in the "ordered phase" ($v_w > 1$), the bottleneck breaks the symmetry of the GP, allowing input geometry to propagate to the output regardless of depth.

Experimental Results

The bottleneck NNGP was evaluated using Marginal Log-Likelihood (MLL) on three distinct datasets:

  1. Rings: A non-linearly separable 3D manifold benchmark consisting of 120 points.
  2. Fisher’s Iris: A classic classification task with correlated labels.
  3. US Census Boston Housing: A standard regression task.

Across all datasets, the model achieved maximum MLL at a finite bottleneck width (e.g., $H=1024$ for Rings, $H=8$ for Iris), outperforming standard infinite-width NNGPs. This suggests that bottlenecks provide a valuable inductive bias rather than just a computational constraint. Furthermore, the authors observed a depth-width tradeoff: as bottlenecks become narrower, the optimal post-bottleneck depth tends to increase.

Conclusion and Future Work

This work formally bridges the gap between NNGP and DGP literatures, showing that DGPs are the literal limit of wide networks with bottlenecks. While the current study focuses on the Bayesian prior and fully-connected layers, it opens the door for future research into "bottleneck Neural Tangent Kernels (NTK)" and extensions to convolutional or attention-based architectures. By preserving information at depth, bottlenecks offer a theoretically grounded mechanism for building more robust deep models.

Summary updated

Abstract

There has recently been much work on the "wide limit" of neural networks, where Bayesian neural networks (BNNs) are shown to converge to a Gaussian process (GP) as all hidden layers are sent to infinite width. However, these results do not apply to architectures that require one or more of the hidden layers to remain narrow. In this paper, we consider the wide limit of BNNs where some hidden layers, called "bottlenecks", are held at finite width. The result is a composition of GPs that we term a "bottleneck neural network Gaussian process" (bottleneck NNGP). Although intuitive, the subtlety of the proof is in showing that the wide limit of a composition of networks is in fact the composition of the limiting GPs. We also analyze theoretically a single-bottleneck NNGP, finding that the bottleneck induces dependence between the outputs of a multi-output network that persists through extreme post-bottleneck depths, and prevents the kernel of the network from losing discriminative power at extreme post-bottleneck depths.
BibTeX
@article{agrawal2020wide,
  author = {Devanshu Agrawal and Theodore Papamarkou and Jacob Hinkle},
  title = {{Wide Neural Networks with Bottlenecks are Deep Gaussian Processes}},
  journal = {JMLR},
  year = {2020},
}
Figure