In standard optimization textbooks, the story of training a neural network is simple: you follow the gradient downhill until it vanishes, eventually coming to rest at a stationary point—a local minimum where the gradient is zero.
However, researchers have observed a persistent gap between this theory and the reality of large-scale deep learning. In practice, even when the training loss has perfectly stabilized, the gradient norm often remains significantly above zero. The weights never actually stop moving. Instead of settling at a single point, the model enters a state of continuous fluctuation.
The Shift from Points to Distributions
If the weights never stop moving, in what sense does the model "converge"? Jingzhao Zhang and Haochuan Li propose a fundamental shift in perspective: we should stop looking for convergence to a single weight vector and start looking for convergence to an invariant measure.
In this framework, the optimization process is viewed as a dynamical system. While the specific values of the weights change at every iteration, the statistical distribution of those weights reaches a steady state. Zhang et al. demonstrate that for large models, weights settle into a stable "orbit" or region of the loss landscape. Within this orbit, the weights fluctuate, but they do so in a way that keeps the overall properties of the model—and the resulting training loss—constant.
Ergodicity and Stable Loss
This perspective explains a common mystery in deep learning: how can the training loss be flat if the weights are still jumping around?
By applying ergodic theory, Haochuan Li, Jingzhao Zhang, Suvrit Sra, and Ali Jadbabaie show that the stability we observe is actually the convergence of the time-average of the loss. Even if the per-iteration loss fluctuates slightly because the weights are moving, the average loss over a window of time becomes extremely stable. In On Convergence of Training Loss Without Reaching Stationary Points, the authors prove that the expected change in loss vanishes as the weight distribution reaches its invariant measure, providing a theoretical foundation for stability that doesn't require the gradient to hit zero.
Why Weights Orbit: Limit Cycles and Asymmetric Noise
Why don't the weights just settle down? The answer lies in the nature of Stochastic Gradient Descent (SGD) and the high-dimensional geometry of neural networks.
Pratik Chaudhari and Stefano Soatto argue that SGD does not just minimize a loss function; it performs implicit variational inference. Because the noise in SGD is "non-isotropic" (lopsided) and the forces are non-conservative, the trajectory of the weights is pushed into limit cycles. Instead of falling into the bottom of a bowl, the weights are caught in a deterministic swirl around critical points.
This behavior is further influenced by the choice of hyperparameters. Kamil Nar and S. Shankar Sastry show that the step size (learning rate) acts as a filter in this dynamical system. It determines which orbits are stable and which are not, effectively selecting the regions of the landscape the model can inhabit.
Beyond Global Objectives
This dynamical systems view is particularly useful for understanding modern training "tricks" that don't fit into standard optimization theory. In non-contrastive self-supervised learning (SSL), techniques like stop-gradients and exponential moving averages (EMA) are used to prevent the model from collapsing.
Jean Ponce, Martial Hebert, and their colleagues show in Dual Perspectives on Non-Contrastive Self-Supervised Learning that these methods often lack a single global objective function to minimize. Instead, they function as stable dynamical systems. Convergence in these models is not about finding a minimum, but about the weights gravitating toward stable equilibria—algebraic varieties—that represent useful, non-collapsed features.
Summary of the Perspective
The transition from "point convergence" to "distributional stability" offers a more accurate map of how neural networks actually learn:
- Weights do not stop: In large models, gradients rarely vanish, and weights continue to move throughout training.
- Distributions stabilize: Training "ends" when the weights reach an invariant measure—a stable probability distribution.
- Loss is a time-average: The perceived stability of the training loss is an ergodic property of the weight distribution.
- Dynamics matter: Concepts like limit cycles and non-conservative forces explain the "swirling" motion of weights that standard optimization ignores.