The Discretization Gap in Neural Network Optimization

565 concepts398 papers

To make the mathematics of deep learning tractable, theorists frequently rely on an elegant simplification: they treat gradient descent as a continuous-time process. By assuming an infinitesimally small learning rate, discrete optimization steps smooth out into continuous trajectories known as gradient flow. Under this lens, researchers can deploy the full toolkit of calculus, proving convergence guarantees, identifying invariant symmetries, and mapping out the implicit biases that steer networks toward low-rank or maximum-margin solutions.

Yet, real-world neural networks are never trained in continuous time. They are optimized using discrete steps with finite, often large, learning rates.

This discrepancy is not just a minor numerical approximation error. The transition from continuous flow to discrete steps fundamentally alters optimization dynamics. The discrete nature of practical optimization introduces systematic errors, breaks theoretical conservation laws, and drives models into entirely different regimes of the loss landscape. Understanding this discretization gap is essential to understanding why deep networks optimize and generalize so effectively in practice.

Discrete Steps as a Hidden Regularizer

When we optimize a model using a finite learning rate, the discrete steps do not perfectly track the continuous gradient path. Instead, they introduce a systematic deviation. Through a mathematical technique called backward error analysis, this deviation can be modeled as a "hidden" explicit penalty added to the original loss function.

In Implicit Gradient Regularization, David G. T. Barrett et al. demonstrated that this implicit penalty is proportional to the squared norm of the gradients. To understand why this happens intuitively, imagine navigating a steep, narrow canyon versus a wide, flat valley:

  • In a steep canyon (high gradients): A discrete step with a finite learning rate will overshoot the canyon floor, climbing up the opposite wall or oscillating wildly. The discrete optimizer "feels" this instability as a high cost.
  • In a flat valley (low gradients): A discrete step lands safely within the valley, smoothly continuing optimization.

Consequently, discrete gradient descent naturally penalizes regions of the parameter space with steep gradients. It acts as an implicit regularizer that steers the model away from sharp local minima and guides it toward "flat minima," which are widely known to generalize better to unseen data. Continuous-time models completely miss this effect because an infinitesimal step never overshoots.

As Dan Zhao notes in Combining Explicit and Implicit Regularization for Efficient Learning in Deep Networks, relying solely on continuous-time proxies can cause theorists to overlook these complex dynamics and failure modes that only emerge in actual discrete implementations, particularly when dealing with adaptive optimizers like Adam.

Breaking Symmetries and Conservation Laws

One of the most striking properties of continuous gradient flow is its preservation of architectural symmetries. For example, in deep homogeneous networks (such as those using ReLU activations or Batch Normalization), gradient flow automatically balances the weights across adjacent layers. As shown by Simon S. Du et al. in Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced, continuous gradient flow preserves the difference between the squared norms of adjacent layers throughout training. If initialized with small weights, the layers remain synchronized in magnitude.

In practice, however, finite learning rates break these conservation laws. In Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics, Daniel Kunin et al. apply finite difference methods to show that discrete steps introduce systematic errors that cause these conserved quantities to drift over time.

Rather than being a destructive flaw, this symmetry breaking can be an optimization asset. In Learning Dynamics of Deep Matrix Factorization Beyond the Edge of Stability, Avrajit Ghosh et al. reveal that when discrete gradient descent breaks these symmetry-induced conservation laws, it actively drives deep linear networks toward flatter minima. Similarly, Ziqing Xu et al. show that while continuous flow predicts stable convergence, the weight imbalance drift introduced by discrete steps can destabilize sub-optimal trajectories, allowing the model to escape local traps and find better global solutions.

Operating at the Edge of Stability

Classical optimization theory dictates that to guarantee stable convergence, the learning rate $\eta$ must be strictly bounded by the inverse of the maximum curvature (the top eigenvalue of the Hessian matrix, $\lambda_{\max}$):

$$\eta < \frac{2}{\lambda_{\max}}$$

If the curvature exceeds this threshold, continuous-time approximations predict that the system will diverge catastrophically.

However, real neural networks defy this boundary. In their seminal paper, Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability, Jeremy Cohen et al. discovered that during discrete gradient descent, neural networks undergo a phenomenon called progressive sharpening. The maximum curvature grows until it hits the exact threshold where $\eta \approx 2/\lambda_{\max}$.

At this point, the optimizer enters the Edge of Stability. Instead of diverging, the training loss begins to oscillate wildly on a micro-scale along the direction of maximum curvature, yet continues to decrease over the long term. This self-regulating, oscillatory behavior is a fundamental characteristic of discrete optimization that cannot exist in smooth, continuous-time gradient flow.

Bridging the Gap Mathematically

Because continuous-time calculus is far easier to analyze than discrete-time difference equations, researchers are actively developing frameworks to incorporate discretization errors directly into continuous equations.

One prominent approach is to introduce corrective terms. In Toward Equation of Motion for Deep Neural Networks: Continuous-time Gradient Descent and Discretization Error Analysis, Taiki Miyagawa derives a formal "Equation of Motion" for neural networks by adding a mathematical counter-term to standard gradient flow equations. This counter-term accounts for the discretization error, allowing the continuous framework to accurately predict the actual weight paths taken during discrete training. Taiki Miyagawa notes that this error is most pronounced during the initial phase of optimization, where the non-smooth nature of the loss landscape near random initialization makes discrete steps deviate sharply from continuous paths.

Another approach focuses on translating convergence proofs. In From Gradient Flow on Population Loss to Learning with Stochastic Gradient Descent, Christopher De Sa et al. develop a "converse Lyapunov" framework. They prove that if a model can be shown to converge under continuous gradient flow, this proof can be used to construct a potential function that guarantees convergence for discrete, noisy stochastic gradient descent (SGD), effectively building a rigorous mathematical bridge between the two regimes.

Ultimately, while continuous-time analysis provides an indispensable baseline for deep learning theory, the true magic of neural network optimization—its ability to find flat minima, break out of sub-optimal symmetries, and navigate chaotic landscapes—often lives entirely within the discretization gap.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 412 papers
  2. Version 1 · 2026-10-04 · 422 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_continuous_discrete_optimization_gap_e787741f,
  title = {continuous-discrete optimization gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/continuous-discrete-optimization-gap/rdr_e787741f5ce246b58b4f69fe4343ac67},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_continuous_discrete_optimization_gap_e787741f_v2,
  title = {continuous-discrete optimization gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/continuous-discrete-optimization-gap/rdr_e787741f5ce246b58b4f69fe4343ac67/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}

See something wrong? Report a mistake