Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup

Sebastian Goldt, Madhu Advani, Andrew M. Saxe, Florent Krzakala, Lenka Zdeborová

Introduction

Classical statistical learning theory suggests a "U-shaped" curve for generalization error: as a model becomes more complex and gains more parameters, it eventually begins to overfit the training noise, causing its performance on unseen data to degrade. However, modern deep learning consistently defies this classical intuition. In practice, massive, over-parameterized models often perform significantly better than smaller ones.

This paper provides a rigorous mathematical framework to investigate this paradox. By analyzing the dynamics of Stochastic Gradient Descent (SGD) through a set of differential equations, the authors demonstrate how and why over-parameterized neural networks can generalize well, revealing that the interplay between network architecture and training dynamics allows larger networks to effectively "denoise" their predictions.

The Teacher-Student Setup & Methods

To isolate the mechanics of generalization, the authors employ the teacher-student setup. In this controlled environment, a "teacher" network generates data, and a "student" network attempts to learn the underlying function from that data.

Diagram of the two-layer neural network architecture used in the teacher-student setup.

The study specifically focuses on the over-parameterized regime, where the student network has more hidden units ($K$) than the teacher network ($M$). The goal is to determine if SGD possesses an "implicit regularization" that prevents these extra units from simply memorizing noise.

The researchers utilize a mix of theoretical physics techniques and rigorous mathematical proofs to map these dynamics:

  • ODE Framework: The authors prove that in the limit of large input dimensions ($N \to \infty$), the discrete steps of SGD can be exactly described by a set of coupled Ordinary Differential Equations (ODEs). These equations track macroscopic "order parameters," such as the overlap between the student and teacher weights.
  • Online Learning: The analysis focuses on "one-pass" SGD, where the network sees each data point only once. This isolates the pure dynamics of the algorithm from the complex effects of re-visiting samples (overfitting to a finite dataset).
  • Perturbative Analysis: To find the final generalization error, they perform a perturbative expansion of their ODEs around the "noiseless" solution, calculating how much the final error increases due to output noise ($ \sigma$).

While the primary analytical results are derived using synthetic Gaussian inputs (allowing for exact calculations), the authors also validated their findings using MNIST images, confirming that the qualitative relationship between network size and error remains consistent.

Key Insights: The Denoising Effect

A critical finding of the paper is that the benefits of over-parameterization depend heavily on how the network is trained and its specific architecture.

The SCM Penalty: In a "Soft Committee Machine" (SCM) setup where only the first layer of the network is trained, extra neurons are actually a liability. Because these "superfluous" units have nothing useful to learn from the teacher, they end up fitting the output noise. Consequently, the generalization error increases linearly with the number of extra units ($L = K - M$).

The Denoising Solution: The narrative changes completely when both layers are trained in sigmoidal networks. Instead of fitting noise, the student network finds a clever workaround. Multiple student neurons "specialize" to mimic the same teacher neuron. Their outputs are then averaged in the second layer, which effectively "denoises" the signal.

Generalization error dynamics showing that larger networks (higher K) achieve lower final error due to the denoising effect.

In this fully trained setup, adding more neurons actually reduces the final generalization error. However, this denoising effect is not universal across all architectures. While sigmoidal networks improve with size, ReLU and linear networks tend to reach a plateau where the error stays constant regardless of how many extra units are added.

Ultimately, these results suggest that "implicit regularization" is not a magic property of the SGD algorithm alone. Rather, it is an emergent property of the interplay between SGD, the specific activation functions, and the training setup.

Key Results & Scaling Laws

The paper formalizes these insights with several key mathematical results:

  • Exact ODEs: Theorem 1.1 provides a rigorous proof that the ODE description of the network's dynamics is asymptotically exact as the input size $N$ grows.
  • SCM Scaling: For sigmoidal and linear SCMs (training only the first layer), the asymptotic generalization error scales as $\epsilon_g^* \sim \eta \sigma^2 L$, where $\eta$ is the learning rate and $L$ is the number of extra units.
  • Two-Layer Sigmoidal Scaling: When training both layers of a sigmoidal network with a single teacher unit ($M=1$), the error scales inversely with the network size: $\epsilon_g^* \sim \frac{1}{K}$. This quantitatively proves that larger networks can perform better.
  • Fixed Points: The ODE framework reveals that while "optimal" configurations exist for linear networks, SGD starting from random weights often fails to find them, instead settling into sub-optimal but stable "denoising" states.

Limitations and Future Work

While this paper provides a robust mathematical foundation for understanding over-parameterization, it has limitations. The study is confined to two-layer networks, and it remains to be seen how these exact dynamics scale to deep architectures with many hidden layers. Furthermore, the analytical reliance on Gaussian inputs and the "one-pass" infinite data assumption glosses over the complexities of real-world, hierarchical data and multi-epoch training.

The authors conclude that a formidable research program is still needed to fully map out how different architectures, deep layers, and complex data distributions interact with SGD to produce the remarkable generalization capabilities observed in modern deep learning.

Summary updated

Abstract

Deep neural networks achieve stellar generalisation even when they have enough parameters to easily fit all their training data. We study this phenomenon by analysing the dynamics and the performance of over-parameterised two-layer neural networks in the teacher-student setup, where one network, the student, is trained on data generated by another network, called the teacher. We show how the dynamics of stochastic gradient descent (SGD) is captured by a set of differential equations and prove that this description is asymptotically exact in the limit of large inputs. Using this framework, we calculate the final generalisation error of student networks that have more parameters than their teachers. We find that the final generalisation error of the student increases with network size when training only the first layer, but stays constant or even decreases with size when training both layers. We show that these different behaviours have their root in the different solutions SGD finds for different activation functions. Our results indicate that achieving good generalisation in neural networks goes beyond the properties of SGD alone and depends on the interplay of at least the algorithm, the model architecture, and the data set.
BibTeX
@inproceedings{goldt2019dynamics,
  author = {Sebastian Goldt and Madhu Advani and Andrew M. Saxe and Florent Krzakala and Lenka Zdeborová},
  title = {{Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
}
Figure