Abstract
Recent theoretical advances have shown that Bayesian neural networks (BNNs) converge to Gaussian processes (GPs) in the infinite-width limit. However, these results typically assume all hidden layers are infinitely wide, neglecting architectures with finite-width "bottlenecks" like autoencoders. This paper proves that BNNs with finite bottlenecks converge to a composition of GPs, termed a bottleneck neural network Gaussian process (bottleneck NNGP). We demonstrate that these bottlenecks prevent deep ReLU networks from losing discriminative power, effectively acting as information anchors that allow the network to "remember" input geometry even at infinite depth.
Introduction: The Wide Limit and its Gaps
The "wide limit" has become a cornerstone of deep learning theory. By treating hidden layers as infinitely wide, researchers can map complex BNNs to Neural Network Gaussian Processes (NNGPs), enabling exact kernel-based analysis. Despite its utility, this framework fails to account for practical architectures that intentionally use narrow layers to force data compression or dimensionality reduction. This paper addresses this gap by asking: what happens when some layers remain narrow while others expand to infinity?
Methodology: The Bottleneck NNGP
The authors formally define the bottleneck NNGP and provide a rigorous mathematical proof that a BNN with finite bottlenecks converges in distribution to a composition of NNGPs—a specific class of Deep Gaussian Process (DGP). A key challenge addressed is proving that the "wide limit of a composition" is equivalent to the "composition of the limiting GPs."
Beyond the convergence proof, the authors analyze the "quadratic correlation"—the correlation between the squares of outputs—induced by the bottleneck. In standard wide networks, outputs are typically independent; however, the bottleneck introduces dependencies that persist through subsequent infinite-width layers.
Theoretical Insights: Information Persistence
A significant finding is the role of bottlenecks as "information anchors." In standard infinite-width ReLU networks, the kernel tends to lose discriminative power as depth increases, eventually treating all inputs as identical. The authors find that a single finite bottleneck prevents this "forgetting."
This persistence of information is uniquely tied to the ReLU activation function. For other common nonlinearities like $tanh$ or $sin$, a bottleneck does not prevent the eventual loss of information in the deep limit. The transition at weight variance $v_w = 1$ acts as a phase transition; in the "ordered phase" ($v_w > 1$), the bottleneck breaks the symmetry of the GP, allowing input geometry to propagate to the output regardless of depth.
Experimental Results
The bottleneck NNGP was evaluated using Marginal Log-Likelihood (MLL) on three distinct datasets:
- Rings: A non-linearly separable 3D manifold benchmark consisting of 120 points.
- Fisher’s Iris: A classic classification task with correlated labels.
- US Census Boston Housing: A standard regression task.
Across all datasets, the model achieved maximum MLL at a finite bottleneck width (e.g., $H=1024$ for Rings, $H=8$ for Iris), outperforming standard infinite-width NNGPs. This suggests that bottlenecks provide a valuable inductive bias rather than just a computational constraint. Furthermore, the authors observed a depth-width tradeoff: as bottlenecks become narrower, the optimal post-bottleneck depth tends to increase.
Conclusion and Future Work
This work formally bridges the gap between NNGP and DGP literatures, showing that DGPs are the literal limit of wide networks with bottlenecks. While the current study focuses on the Bayesian prior and fully-connected layers, it opens the door for future research into "bottleneck Neural Tangent Kernels (NTK)" and extensions to convolutional or attention-based architectures. By preserving information at depth, bottlenecks offer a theoretically grounded mechanism for building more robust deep models.
Summary updated