Collective evolution of weights in wide neural networks

Dmitry YarotskyarXiv 2018 · OpenReview

Abstract

We derive a nonlinear integro-differential transport equation describing collective evolution of weights under gradient descent in large-width neural-network-like models. We characterize stationary points of the evolution and analyze several scenarios where the transport equation can be solved approximately. We test our general method in the special case of linear free-knot splines, and find good agreement between theory and experiment in observations of global optima, stability of stationary points, and convergence rates.

1. Introduction

Training deep neural networks involves navigating a high-dimensional, non-convex loss landscape. While local convergence properties are relatively well-understood, the global behavior—specifically why wide networks consistently find high-quality solutions—remains a central question in deep learning theory. This paper addresses this by shifting the focus from individual parameters to the collective distribution of weights. By treating weights as a continuous "fluid," we derive a macroscopic transport equation that describes their evolution in the large-width limit where the number of neurons $N$ approaches infinity.

2. The Macroscopic Transport Equation

The core of this framework is the derivation of a nonlinear integro-differential equation that serves as a master equation for training dynamics. The approach relies on three key components:

  • Mean-Field Limit: We assume the network is sufficiently wide to represent the weights via a density function $p(\mathbf{w}, t)$ rather than discrete vectors.
  • Continuity Equation: Borrowing from fluid dynamics, we use a continuity equation to model the flow of this density under the influence of gradient descent.
  • Integro-differential Nature: The resulting equation is integro-differential because the velocity of a weight at any point depends on the collective state of the entire distribution (the integral) and the local gradient of the loss function (the differential).

3. Dynamics and Stability of Weight Distributions

A significant insight from this macroscopic view is that the density $p(\mathbf{w}, t)$ does not follow a formal gradient descent on the mean-field loss, even though individual weights do. This discrepancy explains how training can become trapped in stationary points that are not global minima. The stability of these stationary points is crucial; in unstable configurations, even minor initial variances cause weights to diverge along specific axes.

Trajectories of weight parameters $(h, c)$ evolving from initial configurations (black circles) to final stationary states (magenta squares and red stars).

4. Experimental Validation

The theoretical framework was validated using 1D ReLU networks, which function as linear free-knot splines. By approximating synthetic functions such as $f(x) = x^2$ and $f(x) = \sin(k\pi x)$, we compared theoretical predictions against numerical simulations with $N=100$ neurons.

The experiments confirmed the "frequency penalty" phenomenon: convergence speed is inversely proportional to the complexity of the target function. High-frequency components correspond to smaller eigenvalues in the system's operator, leading to significantly slower training rates. Numerical results showed excellent agreement with the theoretical expansion of the mean-field loss.

5. Conclusion and Future Work

This work provides a rigorous macroscopic description of neural network training. While currently limited to shallow architectures and presented at a "physics level of rigor," it offers a path toward predicting convergence and stability without exhaustive simulation. Future research will focus on extending this integro-differential approach to deep architectures and providing a more formal mathematical treatment of the distributional solutions.

Summary updated

Abstract

We derive a nonlinear integro-differential transport equation describing collective evolution of weights under gradient descent in large-width neural-network-like models. We characterize stationary points of the evolution and analyze several scenarios where the transport equation can be solved approximately. We test our general method in the special case of linear free-knot splines, and find good agreement between theory and experiment in observations of global optima, stability of stationary points, and convergence rates.
BibTeX
@misc{yarotsky2018collective,
  author = {Dmitry Yarotsky},
  title = {{Collective evolution of weights in wide neural networks}},
  year = {2018},
  eprint = {1810.03974},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1810.03974},
}
Figure