Overview
Infinite-width neural networks have become a popular theoretical tool in deep learning, but they lack a crucial feature of real-world artificial intelligence: the ability to learn data representations. This summary explores why finite-width "bottlenecks" are essential for model performance and examines a framework that combines the theoretical tractability of infinite networks with the representation-learning power of finite ones.
The Problem with Infinite Width
In recent years, researchers have attempted to understand deep learning by studying the "infinite-width limit," where the number of channels in a network layer approaches infinity. In this limit, neural networks behave like Gaussian Processes (GPs). This equivalence is mathematically elegant and allows for exact Bayesian inference.
However, these infinite-width Bayesian neural networks (BNNs) often perform significantly worse than standard finite-width networks trained with Stochastic Gradient Descent (SGD). The core issue is that infinite networks possess a fixed kernel. Their internal mechanism for measuring similarity between data points is determined entirely by architectural hyperparameters (such as depth) and cannot adapt to the training data. Consequently, infinite networks cannot perform representation learning—the fundamental mechanism that enables deep learning models to excel at complex tasks like transfer learning.
Methodology
To bridge the gap between theoretical infinite models and practical finite models, the researchers employed several analytical and empirical approaches:
- Toy Linear Models: The authors analyzed a simple two-layer linear network to demonstrate how finite width provides representational "flexibility" that vanishes as the network becomes infinitely wide.
- Infinite Networks with Bottlenecks: They introduced a novel architecture where layers alternate between finite-width "bottlenecks" and infinite-width expansions. This design maintains mathematical tractability while restoring the network's ability to learn representations through the finite layers.
- Prior and Posterior Analysis: Analytic results were derived to show how "kernel flexibility" (the variance of the kernel under the prior) scales with network width and depth.
- Empirical Validation: The internal representations of a standard 34-layer ResNet trained on CIFAR-10 were compared against both infinite-width models and the proposed finite-width theoretical predictions.
Key Insights and Results
The "Infinite" Trap and the Need for Flexibility
In a truly infinite network, the kernel is "frozen." It cannot learn, for instance, that two visually distinct images of the same object class should be treated as similar in the feature space. This rigidity explains why transfer learning fails in infinite BNNs: a representation fixed by hyperparameters cannot be fine-tuned.
Conversely, finite width introduces variability into the kernel. While sometimes viewed merely as "noise," this variability is precisely what allows the training data to shape the representation. Toy experiments on synthetic data reveal a "sweet spot" for network width; making a network too wide can actually degrade performance because the model loses the flexibility required to overcome initial model mismatch.
Width vs. Depth: The Mechanics of Kernel Variance
The authors mathematically prove that kernel variance—a proxy for representational flexibility—is proportional to network depth ($L+1$) and inversely proportional to width ($N$). As width increases, the kernel "concentrates" on its mean, becoming increasingly rigid.
Real-World Representation Shift in ResNets
Remarkably, the representations learned by state-of-the-art ResNets behave much more like the authors' finite-width linear models than theoretical infinite-width GPs.
As training progresses, the ResNet's internal kernel actively moves away from the "infinite" prior. Its correlation with the infinite kernel drops toward zero, while its correlation with the "output kernel" (the target labels) steadily increases. This divergence confirms that active representation learning is occurring—a process that infinite models fundamentally cannot replicate.
Limitations and Future Directions
The theoretical analysis in this work relies heavily on linear networks to maintain mathematical tractability. While empirical results show these findings hold surprisingly well for non-linear architectures like ResNets, developing a full non-linear theory for finite-width representation learning remains an open challenge.
Furthermore, Maximum A-Posteriori (MAP) inference in finite networks can be problematic, potentially leading to overfitting or underfitting depending on the specific width. Future research may explore more efficient methods for performing full Bayesian sampling in these "bottlenecked" infinite networks, potentially closing the performance gap with standard SGD-trained models.
Summary updated