A note on Linear Bottleneck networks and their Transition to Multilinearity

Libin Zhu, Parthe Pandit, Mikhail Belkin

TL;DR

While infinitely wide neural networks are known to behave like simple linear functions of their weights during training (the Neural Tangent Kernel regime), introducing even a single narrow "bottleneck" layer completely breaks this property. This paper proves that linear networks with $B-1$ bottleneck layers instead transition to a degree-$B$ multilinear function of their weights as the non-bottleneck layers become infinitely wide. Remarkably, this polynomial degree depends solely on the number of bottlenecks, remaining completely independent of the network's total depth.

What Problem Does This Paper Address?

The Neural Tangent Kernel (NTK) framework revolutionized our theoretical understanding of deep learning by showing that ultra-wide neural networks behave like linear models in their parameters during gradient descent. However, a strict mathematical requirement for this "transition to linearity" is that every single layer must be infinitely wide.

In practice, practical architectures—like autoencoders or modern residual networks—frequently employ narrow "bottleneck" layers to compress representations or reduce computation. Previous work demonstrated that adding even one narrow layer prevents the network from transitioning to a linear model. This leaves a major theoretical gap: if wide networks with bottlenecks don't become linear, what kind of functions do they represent near initialization?

This paper answers that question for linear neural networks (networks with identity activations), providing a stepping stone toward understanding more complex, non-linear bottleneck architectures.

Methods

The authors analyze fixed-depth linear networks where $B-1$ bottleneck layers have a fixed, narrow width $r = O(1)$, while the width $m$ of all other layers approaches infinity.

To determine the functional form of these networks, the authors use a Taylor expansion with a Lagrange remainder term (specifically analyzing the $n$-jet of the network function) combined with the Mean Value Theorem. Their key technical strategy involves bounding the spectral norms of the network's higher-order derivatives with respect to its weights:

  • They prove that the spectral norm of the $(B+1)$-th derivative scales as $O(1/\sqrt{m})$, meaning it vanishes entirely as the non-bottleneck width $m$ goes to infinity.
  • Meanwhile, they show the spectral norm of the $B$-th derivative is bounded below by $\Omega(1)$, ensuring it does not vanish.

This rigorously establishes that the network function reduces exactly to a $B$-th degree polynomial. They further prove that this polynomial is multilinear, meaning the network is jointly linear in the blocks of weights separated by the bottlenecks.

Datasets

As a purely theoretical paper, no standard empirical datasets (like ImageNet or MNIST) were used. The authors validate their theoretical claims using synthetic numerical simulations, tracking the network's output as weights are perturbed along random 1D subspaces.

Key Insights and Ideas

  • Depth Independence: The most striking insight is that the complexity (degree) of the resulting function depends only on the number of bottleneck layers, not the total depth of the network. For example, a 100-layer linear network with only 1 bottleneck layer will still behave precisely like a bilinear (degree-2) function near initialization.
  • Drastic Complexity Reduction: A standard linear network is technically a polynomial of its weights with a degree equal to its total number of layers. The paper reveals that making non-bottleneck layers infinitely wide drastically collapses this complexity, reducing the effective polynomial degree down to the number of bottlenecks plus one.
  • The Role of Linear Activations: The transition to a clean multilinear model relies heavily on linear activations. As shown in the paper's visualizations, cascading wide layers through a bottleneck with non-linear activations (like tanh) results in an inflection point, proving it is not a simple quadratic function.

Key Results

  • Theorem 1 (Transition to Bilinearity): For a 4-layer linear network with 1 bottleneck layer, the network function $g(W; x)$ transitions to a bilinear function $q(W)$ as width $m \to \infty$. The approximation error scales as $O(1/\sqrt{m})$ with a high probability of at least $1 - 12e^{-m/32}$.
  • Theorem 2 (Transition to Multilinearity): Generalizing this, a network with $B-1$ bottlenecks transitions to a $B$-th degree multilinear function of the weight blocks between consecutive bottlenecks, with the remainder term similarly vanishing at a rate of $O(1/\sqrt{m})$.
  • Empirical Verification: The authors illustrate these transitions by plotting the network function along random parameter directions $\Delta$.
Figure 1: Network function behavior along a 1D subspace for different activations and weight perturbations.

As seen in the visual experiments above:

  • (a) A tanh activation network shows non-quadratic behavior.
  • (b) An identity activation network shows clean quadratic behavior.
  • (c) & (d) Restricting perturbations to weights either entirely before ($W_1$) or entirely after ($W_2$) the bottleneck results in perfect linearity, confirming the bilinear nature of the network.

Limitations and Future Work

  • Strictly Linear Networks: The core proofs apply only to networks with linear activations. While the authors suspect a similar multilinear transition might hold if only the very first block of the network contains non-linear activations, extending this theory to fully non-linear deep bottleneck networks remains an open problem.
  • Optimization Dynamics: While the paper successfully maps bottleneck networks to well-studied multilinear models, it does not explicitly analyze how this multilinear structure impacts the actual convergence speed or implicit regularization of gradient descent during training.

Summary updated

Abstract

Randomly initialized wide neural networks transition to linear functions of weights as the width grows, in a ball of radius $O(1)$ around initialization. A necessary condition for this result is that all layers of the network are wide enough, i.e., all widths tend to infinity. However, the transition to linearity breaks down when this infinite width assumption is violated. In this work we show that linear networks with a bottleneck layer learn bilinear functions of the weights, in a ball of radius $O(1)$ around initialization. In general, for $B-1$ bottleneck layers, the network is a degree $B$ multilinear function of weights. Importantly, the degree only depends on the number of bottlenecks and not the total depth of the network.
BibTeX
@misc{zhu2022note,
  author = {Libin Zhu and Parthe Pandit and Mikhail Belkin},
  title = {{A note on Linear Bottleneck networks and their Transition to Multilinearity}},
  year = {2022},
  eprint = {2206.15058},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2206.15058},
}
Figure