TL;DR
While infinitely wide neural networks are known to behave like simple linear functions of their weights during training (the Neural Tangent Kernel regime), introducing even a single narrow "bottleneck" layer completely breaks this property. This paper proves that linear networks with $B-1$ bottleneck layers instead transition to a degree-$B$ multilinear function of their weights as the non-bottleneck layers become infinitely wide. Remarkably, this polynomial degree depends solely on the number of bottlenecks, remaining completely independent of the network's total depth.
What Problem Does This Paper Address?
The Neural Tangent Kernel (NTK) framework revolutionized our theoretical understanding of deep learning by showing that ultra-wide neural networks behave like linear models in their parameters during gradient descent. However, a strict mathematical requirement for this "transition to linearity" is that every single layer must be infinitely wide.
In practice, practical architectures—like autoencoders or modern residual networks—frequently employ narrow "bottleneck" layers to compress representations or reduce computation. Previous work demonstrated that adding even one narrow layer prevents the network from transitioning to a linear model. This leaves a major theoretical gap: if wide networks with bottlenecks don't become linear, what kind of functions do they represent near initialization?
This paper answers that question for linear neural networks (networks with identity activations), providing a stepping stone toward understanding more complex, non-linear bottleneck architectures.
Methods
The authors analyze fixed-depth linear networks where $B-1$ bottleneck layers have a fixed, narrow width $r = O(1)$, while the width $m$ of all other layers approaches infinity.
To determine the functional form of these networks, the authors use a Taylor expansion with a Lagrange remainder term (specifically analyzing the $n$-jet of the network function) combined with the Mean Value Theorem. Their key technical strategy involves bounding the spectral norms of the network's higher-order derivatives with respect to its weights:
- They prove that the spectral norm of the $(B+1)$-th derivative scales as $O(1/\sqrt{m})$, meaning it vanishes entirely as the non-bottleneck width $m$ goes to infinity.
- Meanwhile, they show the spectral norm of the $B$-th derivative is bounded below by $\Omega(1)$, ensuring it does not vanish.
This rigorously establishes that the network function reduces exactly to a $B$-th degree polynomial. They further prove that this polynomial is multilinear, meaning the network is jointly linear in the blocks of weights separated by the bottlenecks.
Datasets
As a purely theoretical paper, no standard empirical datasets (like ImageNet or MNIST) were used. The authors validate their theoretical claims using synthetic numerical simulations, tracking the network's output as weights are perturbed along random 1D subspaces.
Key Insights and Ideas
- Depth Independence: The most striking insight is that the complexity (degree) of the resulting function depends only on the number of bottleneck layers, not the total depth of the network. For example, a 100-layer linear network with only 1 bottleneck layer will still behave precisely like a bilinear (degree-2) function near initialization.
- Drastic Complexity Reduction: A standard linear network is technically a polynomial of its weights with a degree equal to its total number of layers. The paper reveals that making non-bottleneck layers infinitely wide drastically collapses this complexity, reducing the effective polynomial degree down to the number of bottlenecks plus one.
- The Role of Linear Activations: The transition to a clean multilinear model relies heavily on linear activations. As shown in the paper's visualizations, cascading wide layers through a bottleneck with non-linear activations (like
tanh) results in an inflection point, proving it is not a simple quadratic function.
Key Results
- Theorem 1 (Transition to Bilinearity): For a 4-layer linear network with 1 bottleneck layer, the network function $g(W; x)$ transitions to a bilinear function $q(W)$ as width $m \to \infty$. The approximation error scales as $O(1/\sqrt{m})$ with a high probability of at least $1 - 12e^{-m/32}$.
- Theorem 2 (Transition to Multilinearity): Generalizing this, a network with $B-1$ bottlenecks transitions to a $B$-th degree multilinear function of the weight blocks between consecutive bottlenecks, with the remainder term similarly vanishing at a rate of $O(1/\sqrt{m})$.
- Empirical Verification: The authors illustrate these transitions by plotting the network function along random parameter directions $\Delta$.
As seen in the visual experiments above:
- (a) A
tanhactivation network shows non-quadratic behavior. - (b) An identity activation network shows clean quadratic behavior.
- (c) & (d) Restricting perturbations to weights either entirely before ($W_1$) or entirely after ($W_2$) the bottleneck results in perfect linearity, confirming the bilinear nature of the network.
Limitations and Future Work
- Strictly Linear Networks: The core proofs apply only to networks with linear activations. While the authors suspect a similar multilinear transition might hold if only the very first block of the network contains non-linear activations, extending this theory to fully non-linear deep bottleneck networks remains an open problem.
- Optimization Dynamics: While the paper successfully maps bottleneck networks to well-studied multilinear models, it does not explicitly analyze how this multilinear structure impacts the actual convergence speed or implicit regularization of gradient descent during training.
Summary updated