In theoretical deep learning, we often simplify the chaotic process of training a neural network by imagining it as a smooth, continuous slide down a hill. This mathematical idealization, known as gradient flow, assumes that we update weights using infinitesimal steps. It allows researchers to use the elegant tools of calculus and differential equations to prove that networks will converge.
However, real-world training never uses infinitesimal steps. We use gradient descent (GD) with finite, often large, learning rates. For years, this gap was treated as a mere implementation detail—a "discretization error" that would vanish if the learning rate were small enough.
Recent research has flipped this perspective. Instead of a bug, the discrete nature of gradient descent is a feature. The "drift" caused by finite steps acts as a hidden force—an implicit regularizer—that steers models toward solutions that generalize better than the ones found by continuous flow.
The Geometry of the Gap
The fundamental tension lies in how we model the trajectory of weights. Yacine Chitour, Zhenyu Liao, and Romain Couillet demonstrated in A Geometric Approach of Gradient Descent Algorithms in Neural Networks that continuous-time gradient flow in linear networks almost always avoids saddle points and finds global minima.
But as Gabin Maxime Nguegnang, Holger Rauhut, and Ulrich Terstiege point out in Convergence of gradient descent for learning linear neural networks, the symmetries and invariants that hold in continuous time can "drift" or break when steps are discrete. This drift isn't just noise; it changes the destination.
Neural Mechanics and Kinetic Symmetry Breaking
To understand why this drift matters, researchers have turned to the language of physics. In Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics, authors including Daniel Kunin, Surya Ganguli, and Hidenori Tanaka treat the training process as a physical system governed by conservation laws.
They found that while the loss function might have certain symmetries (like scaling weights in one layer up and another down without changing the output), the learning rule itself can break these symmetries. Daniel Kunin and Hidenori Tanaka call this Kinetic Symmetry Breaking (KSB) in Noether's Learning Dynamics: The Role of Kinetic Symmetry Breaking in Deep Learning. This phenomenon explains why layers in a deep network automatically stay "balanced" in magnitude—the discrete updates push the system toward a state where the kinetic energy of the weights is minimized.
Implicit Gradient Regularization
If discrete steps introduce a bias, what exactly are they regularizing? David G. T. Barrett and Benoit Dherin provided a breakthrough answer using "backward error analysis" in Implicit Gradient Regularization.
They showed that discrete gradient descent on a loss function $L$ is mathematically equivalent to continuous gradient flow on a modified loss function: $$\tilde{L} = L + \frac{\eta}{4} \|\nabla L\|^2$$ where $\eta$ is the learning rate. This means that by taking discrete steps, the optimizer is implicitly penalizing the norm of the gradient. It doesn't just want to find a minimum; it wants to find a flat minimum—a region where the gradient doesn't change rapidly. Flat minima are widely believed to generalize better because they are robust to small perturbations in the data or parameters.
The Edge of Stability and Principal Flow
This discrete perspective also explains why training often feels "messy." In On a continuous time model of gradient descent dynamics and instability in deep learning, Benoit Dherin, Chongli Qin, and colleagues introduced the Principal Flow (PF).
Unlike standard gradient flow, the PF explicitly incorporates the learning rate into the differential equations. This allows it to model the "Edge of Stability"—a phenomenon where the optimizer operates at the very brink of divergence, oscillating wildly but ultimately finding better solutions. This understanding led to the development of the Drift Adjusted Learning rate (DAL), an optimizer that self-tunes by accounting for this discretization drift.
Why It Works: Benign Overfitting
The ultimate goal of this research direction is to explain why deep networks generalize even when they have enough parameters to memorize the training data perfectly. Ichiro Hashimoto proved in Directional Convergence, Benign Overfitting of Gradient Descent in leaky ReLU two-layer Neural Networks that discrete gradient descent (not just the flow) leads to "benign overfitting."
Similarly, Tomaso Poggio and Qianli Liao argue in Explicit regularization and implicit bias in deep network classifiers trained with the square loss that the combination of discrete updates and normalization creates a bias toward "minimum norm" solutions.
By moving away from the "smooth" ideal of continuous flow and embracing the "jitter" of discrete steps, these researchers are uncovering the mechanical reasons why deep learning is so effective: the algorithm itself is a regularizer.