Implicit Acceleration and Feature Learning in Infinitely Wide Neural Networks with Bottlenecks

Etai Littwin, Omid Saremi, Shuangfei Zhai, Vimal Thilak, Hanlin Goh, Joshua M. Susskind, Greg YangarXiv 2021 · OpenReview

Abstract

We analyze the learning dynamics of infinitely wide neural networks with a finite sized bottleneck. Unlike the neural tangent kernel limit, a bottleneck in an otherwise infinite width network allows data-dependent feature learning in its bottleneck representation. We empirically show that a single bottleneck in infinite networks dramatically accelerates training when compared to purely infinite networks, with an improved overall performance. We discuss the acceleration phenomena by drawing similarities to infinitely wide deep linear models, where the acceleration effect of a bottleneck can be understood theoretically.

Introduction

In deep learning theory, "infinitely wide" neural networks are a popular tool because their behavior becomes mathematically predictable. Usually, these models fall into the Neural Tangent Kernel (NTK) regime, where the network evolves like a simple linear model during training. While easy to study, this regime has a major flaw: the network stops "learning features" from data; its internal representations stay nearly frozen.

Furthermore, standard infinite-width limits assume all layers are infinite. However, many successful real-world architectures, like autoencoders, use narrow "bottleneck" layers to force the model to learn compressed, meaningful representations. Standard theory effectively "assumes away" these bottlenecks. This paper investigates what happens to the learning dynamics when a finite bottleneck is integrated into an infinite network.

Methodology

The authors analyze a composition of two networks: an infinite-width encoder ($g$) that maps inputs to a finite $d$-dimensional bottleneck, and an infinite-width decoder ($f$) that produces the final output.

Key methodological contributions include:

  • Functional Dynamics: Instead of tracking millions of individual weights, the authors derive a set of Ordinary Differential Equations (ODEs) that describe how the network's output and its internal "bottleneck" representations evolve in function space.
  • Jacobian Analysis: They prove that the input-output Jacobian of the network (how sensitive the output is to the bottleneck) behaves like a Gaussian Process at initialization and evolves linearly during training.
  • Linear Model Equivalence: To explain why bottlenecks help, they draw theoretical parallels to "deep linear models," where stacking layers with a bottleneck is known to create an "acceleration effect" similar to momentum.

Results and Discussion

The primary finding is that the presence of a finite bottleneck allows for feature learning, breaking the static nature of the standard NTK regime. The intermediate activations change in a non-trivial, data-dependent way during training.

The Acceleration Phenomenon

Surprisingly, adding a bottleneck makes the network train faster. In purely infinite networks, training speed is limited by the kernel; adding a bottleneck introduces an "implicit acceleration" that mimics the effects of momentum without needing an explicit momentum parameter.

Training loss comparison showing that finite bottlenecks (widths 50-1000) significantly accelerate convergence compared to the infinite-width limit (inf).

As seen in Figure 1, the "inf" baseline (purely infinite network) exhibits a much slower decay in loss. In contrast, networks with finite bottlenecks show a rapid initial drop. There is an "optimal" bottleneck size; if a bottleneck is too wide, the acceleration effect diminishes, approaching the infinite limit.

Extended training dynamics demonstrating that the acceleration effect persists over many steps, with narrower bottlenecks (e.g., width 20) achieving the fastest convergence and lowest loss.

Beyond speed, bottlenecked models often achieved better final test performance. On synthetic datasets, the infinite-width model had the highest test loss, while models with bottlenecks of size 20–500 performed significantly better. The authors' derived ODEs almost perfectly match the actual training trajectories of large-but-finite networks (e.g., width 100,000), confirming the theoretical model's accuracy.

Limitations and Future Work

The rigorous mathematical analysis is primarily focused on "shallow" setups with one hidden layer before and after the bottleneck. Extending this to deeper architectures involves significant technical challenges regarding the structure of the Jacobian. Additionally, while the theory assumes smooth activations, experiments using ReLU (via a softplus approximation) suggest the findings are robust, though a direct treatment of non-smooth activations remains for future work. Finally, the study focuses on Gradient Flow; the discrete-time effects of large learning rates in standard SGD were not the primary focus.

Summary updated

Abstract

We analyze the learning dynamics of infinitely wide neural networks with a finite sized bottle-neck. Unlike the neural tangent kernel limit, a bottleneck in an otherwise infinite width network al-lows data dependent feature learning in its bottle-neck representation. We empirically show that a single bottleneck in infinite networks dramatically accelerates training when compared to purely in-finite networks, with an improved overall performance. We discuss the acceleration phenomena by drawing similarities to infinitely wide deep linear models, where the acceleration effect of a bottleneck can be understood theoretically.
BibTeX
@misc{littwin2021implicit,
  author = {Etai Littwin and Omid Saremi and Shuangfei Zhai and Vimal Thilak and Hanlin Goh and Joshua M. Susskind and Greg Yang},
  title = {{Implicit Acceleration and Feature Learning in Infinitely Wide Neural Networks with Bottlenecks}},
  year = {2021},
  eprint = {2107.00364},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2107.00364},
}
Figure