Abstract
Recent work has argued that neural networks can be understood theoretically by taking the number of channels to infinity, at which point the outputs become Gaussian process (GP) distributed. However, we note that infinite Bayesian neural networks lack a key facet of the behaviour of real neural networks: the fixed kernel, determined only by network hyperparameters, implies that they cannot do any form of representation learning. The lack of representation or equivalently kernel learning leads to less flexibility and hence worse performance, giving a potential explanation for the inferior performance of infinite networks observed in the literature. We give analytic results characterising the prior over representations and representation learning in finite deep linear networks. We show empirically that the representations in SOTA architectures such as ResNets trained with SGD are much closer to those suggested by our deep linear results than by the corresponding infinite network. This motivates the introduction of a new class of network: infinite networks with bottlenecks, which inherit the theoretical tractability of infinite networks while at the same time allowing representation learning.
Introduction
In deep learning theory, the "infinite-width limit" is often treated as a gold standard for mathematical elegance. As the number of channels in a layer goes to infinity, the network's output distribution converges to a Gaussian Process (GP). While this provides a powerful framework for Bayesian analysis, it introduces a fundamental flaw: infinite Bayesian neural networks (BNNs) have a fixed kernel.
In a standard finite neural network, the "representation" (the features the network extracts) adapts to the data during training. In contrast, the top-layer representation of an infinite BNN is determined entirely by the architecture and hyperparameters, rendering it "blind" to actual data labels. This lack of flexibility explains the performance gap often observed between theoretically elegant infinite models and their finite, "messy" counterparts.
The Inflexibility of Infinity
In a finite network, the kernel is a random variable that the data can shape. As width increases, this variance vanishes, and the kernel becomes a deterministic constant. The author identifies that kernel flexibility (variance) is proportional to depth but inversely proportional to width ($L/N$). This suggests that deep, narrow networks are inherently better at representation learning than shallow, wide ones.
Furthermore, weight sharing in Convolutional Neural Networks (CNNs) actually increases kernel flexibility compared to Locally Connected Networks (LCNs), providing a theoretical justification for the empirical success of CNNs in image processing.
Performance Degradation and the Optimal Width
Empirical evidence suggests that increasing width beyond a certain point can be counterproductive. In experiments with model mismatch, both Bayesian model evidence and predictive log-probability peak at a finite number of hidden units before declining.
This degradation occurs because the network loses its ability to adapt its representation as it approaches the infinite-width limit, where the kernel becomes fixed and non-adaptive.
Representation Learning in ResNets
To validate these theoretical insights, the author compared the kernels of finite ResNet-34 models trained with Stochastic Gradient Descent (SGD) against their theoretical infinite-width counterparts on CIFAR-10. The results show that trained ResNets behave fundamentally differently than infinite networks.
As shown in Figure 4, training shifts the representation from the "input kernel" (raw input similarity) toward the "output kernel" (label similarity). Infinite networks remain stuck at the input kernel, unable to perform this posterior shift. Additionally, the eigenvalue spectrum of trained finite kernels remains flatter in intermediate layers, suggesting the network preserves information before collapsing it to class labels at the final layer.
Conclusion: Infinite Networks with Bottlenecks
To bridge the gap between theoretical tractability and empirical performance, the author proposes infinite networks with bottlenecks. By alternating between infinitely wide layers and finite "bottleneck" layers, these architectures force information compression and re-introduce the ability to learn representations while maintaining the mathematical advantages of the infinite limit. This framework offers a promising path forward for designing theoretically grounded yet high-performing neural architectures.
Summary updated