TL;DR
This paper bridges the gap between two theoretical frameworks for understanding neural networks: Gaussian Processes (GPs) in the "wide limit" and Deep Gaussian Processes (DGPs). The authors prove that neural networks with "bottleneck" layers (finite-width layers surrounded by infinite-width layers) converge to DGPs, specifically compositions of GPs. This work provides a theoretical foundation for understanding deep neural networks with architectures that deviate from the purely infinite-width assumption, offering new insights into their behavior and potential.
What Problem Does This Paper Address?
The theoretical understanding of deep neural networks (DNNs) has seen significant progress through the lens of Gaussian Processes (GPs). Specifically, when all hidden layers of a Bayesian Neural Network (BNN) are taken to infinite width, the network converges to a Neural Network Gaussian Process (NNGP). However, this theoretical framework doesn't directly apply to architectures that intentionally use finite-width layers, often called "bottlenecks." These bottlenecks are common in practical architectures like autoencoders and word embedding layers, where they are crucial for learning compressed representations. This paper addresses the challenge of theoretically understanding BNNs with such bottleneck layers, aiming to connect them to the existing GP framework, particularly Deep Gaussian Processes (DGPs).
Methods
The core of this paper is a formal proof that a sequence of BNNs with bottleneck layers, as their infinite-width components grow, converges to a Deep Gaussian Process (DGP) which is a composition of NNGPs. This is achieved by:
- Generalizing the NNGP Theorem: The authors extend the existing NNGP theorem (which deals with all layers going to infinite width) to handle architectures with finite-width bottleneck layers.
- Composition of GPs: They demonstrate that the limit of a composition of BNNs is the composition of their limiting GPs. This is a non-trivial result that requires careful justification.
- Theoretical Analysis of Bottleneck Effects: The paper analyzes the impact of a bottleneck layer on the network's behavior, particularly focusing on:
- Quadratic Correlation: Investigating how bottlenecks induce dependence between outputs, even when linear correlations remain zero.
- Deep Limit Behavior: Analyzing the network's behavior as the depth after the bottleneck goes to infinity, revealing phase transitions and the preservation of discriminative power in certain regimes.
- Empirical Validation: The theoretical findings are supported by experiments on simulated and real-world datasets, demonstrating the utility of bottleneck NNGPs and their relationship to no-bottleneck NNGPs.
Datasets
The paper uses the following datasets for empirical evaluation:
- Simulated Rings Dataset: A non-linearly separable binary classification problem designed to test the model's ability to handle complex data manifolds.
- Fisher's Iris Dataset: A classic multi-class classification dataset.
- US Census Boston Housing Prices Dataset: A regression task.
For all datasets, the authors standardized the input and target sets to ensure comparable scales.
Key Insights and Ideas
- Bottlenecks as DGP Components: The most significant insight is the formal proof that neural networks with bottlenecks, when taken to the wide limit, behave as Deep Gaussian Processes (DGPs) composed of NNGPs. This unifies two important theoretical frameworks.
- Bottlenecks Induce Non-trivial Correlations: Even though linear correlations between outputs of a bottleneck NNGP remain zero, quadratic correlations can be non-zero. This suggests a subtle but important form of dependence induced by bottlenecks.
- Preserving Discriminative Power at Depth: Unlike standard NNGPs with ReLU activations that degenerate at extreme depths, bottleneck NNGPs can retain discriminative power. This is a crucial finding for understanding very deep networks.
- Phase Transitions at Bottleneck Widths: The behavior of bottleneck NNGPs exhibits phase transitions related to the variance of the weight prior, particularly around $v_w = 1$. This suggests that optimal network configurations might lie near these critical points.
- Bottlenecks Enable Deeper Architectures: The theoretical analysis highlights that bottlenecks are fundamental for enabling networks to "go deeper" while preserving information and discriminative power.
Key Results
- Theorem 4 (Bottleneck NNGP Theorem): Establishes that a sequence of bottleneck BNNs converges in distribution to a bottleneck NNGP, which is a composition of NNGPs.
- Theorem 8 (Wide Bottleneck Correspondence): Shows that as bottleneck width increases to infinity, a bottleneck NNGP converges to a no-bottleneck NNGP.
- Empirical MLL Peaks: Experiments show that the marginal log-likelihood (MLL) of bottleneck NNGP architectures peaks at finite bottleneck widths and post-bottleneck depths, indicating that these parameters are crucial for model performance.
- Quadratic Correlation Analysis: The paper provides analytical expressions and empirical verification for quadratic correlations, demonstrating their dependence on bottleneck width, depth, and prior variances.
- Phase Transition at $v_w=1$: The infinite-depth quadratic correlation exhibits a phase transition at $v_w=1$, suggesting a critical point for network behavior.
Limitations and Future Work
- Focus on ReLU Activation: While the paper briefly considers other nonlinearities, the primary theoretical analysis focuses on the ReLU activation. Extending these findings to a broader range of activations would be valuable.
- Extension to Convolutional Architectures: The authors postulate that their main theorem can be extended to convolutional architectures, which is a promising direction for future work.
- Implications for Learning Dynamics: The paper suggests investigating the implications of bottleneck NNGPs for the learning dynamics of deterministic neural networks (DNNs), particularly through the lens of the neural tangent kernel (NTK).
- Practical BNN Inference: The authors intend to explore the practicality of using non-parametric DGP-based approaches for BNN inference, especially in the context of finite-width networks.
- Bottleneck NTK: Investigating the "bottleneck NTK" for finite-width DNNs is proposed as future work, which could offer a more refined description of DNN evolution.
Summary updated