Backward Error Analysis for Neural Optimization

This direction is merged into another direction: continuous-discrete optimization gap. Its last version stays readable and citable here.

Version 1 of 119 papers19 conceptsfrozen 2026-10-04

In textbook deep learning, we often treat Gradient Descent (GD) as a discrete approximation of a continuous "gradient flow." We imagine the model sliding smoothly down the surface of a loss function $L(\theta)$. However, in practice, we don't slide; we jump. We take finite steps with a learning rate $\eta > 0$.

Recent research reveals that these discrete jumps do not simply follow the original loss. Instead, the optimizer implicitly minimizes a modified objective function. This phenomenon, analyzed through a technique called Backward Error Analysis (BEA), explains why the specific choice of optimizer, learning rate, and batch size fundamentally changes the "quality" of the minima a model finds.

The Gap Between Flow and Steps

When we analyze optimization using calculus, we often look at the Ordinary Differential Equation (ODE) $\dot{\theta} = -\nabla L(\theta)$. This is the "ideal" path. But an actual update $\theta_{t+1} = \theta_t - \eta \nabla L(\theta_t)$ deviates from this path.

As Benoit Dherin and David G. T. Barrett demonstrate in Implicit Gradient Regularization, we can use BEA to find a new, modified ODE that the discrete steps follow much more closely. By Taylor-expanding the discrete update rule, they show that Gradient Descent is actually performing continuous gradient flow on a modified loss:

$$\tilde{L}(\theta) = L(\theta) + \frac{\eta}{4} \|\nabla L(\theta)\|^2$$

This extra term is the implicit regularizer. Because the optimizer is trying to minimize $\tilde{L}$, it doesn't just look for low values of $L$; it also seeks regions where the gradient norm $\|\nabla L(\theta)\|^2$ is small. This naturally pushes the model toward "flat minima," which are widely believed to generalize better to unseen data.

Why SGD and Adam Generalize Differently

The power of BEA lies in its ability to differentiate between optimizers that might look similar on paper but behave differently in practice.

In On the Origin of Implicit Regularization in Stochastic Gradient Descent, Samuel L. Smith and colleagues extend this analysis to the stochastic setting. They find that the noise from minibatches introduces an additional regularization term proportional to the ratio of the learning rate to the batch size ($\eta/B$). This provides a rigorous mathematical footing for the common heuristic that increasing the learning rate or decreasing the batch size can improve generalization.

Conversely, BEA helps explain the "generalization gap" of adaptive optimizers. Matias D. Cattaneo and Jason M. Klusowski show in On the Implicit Bias of Adam that while SGD penalizes the $L_2$ norm of gradients, Adam can actually anti-penalize a perturbed $L_1$ norm. This "anti-regularization" drives the model toward sharper, less robust regions of the loss landscape, explaining why Adam often requires more careful tuning or explicit weight decay to match SGD's performance.

Dynamics Beyond the Minimum

BEA isn't just about where the model ends up; it's about how it moves. In On a continuous time model of gradient descent dynamics and instability in deep learning, Benoit Dherin and Mihaela Rosca use BEA to model the "Edge of Stability." This is the regime where the learning rate is so high that the loss oscillates, yet the model still converges. By deriving a "Principal Flow," they capture these oscillations in a continuous-time framework, leading to more stable optimizers like DAL (Drift Adjusted Learning rate).

This perspective is also vital for complex training setups like Generative Adversarial Networks (GANs). In Discretization Drift in Two-Player Games, the authors show that the discrete nature of updates in a two-player game creates a "drift" that can cause the system to spiral away from the equilibrium. BEA allows researchers to identify these drift terms and cancel them out, stabilizing training.

Stochasticity and Stability

While deterministic BEA uses standard ODEs, stochastic optimizers require a more sophisticated bridge. Desmond Higham and Stefano Di Giovacchino have developed Stochastic Backward Error Analysis, which maps discrete stochastic updates to modified Stochastic Differential Equations (SDEs).

Similarly, Xiang Li and Niao He use a Hessian-aware SDE to explain how SGD escapes saddle points. By accounting for how discretization noise interacts with the curvature (Hessian) of the loss, they show that the "messiness" of discrete steps is exactly what allows models to find superior solutions that a perfectly smooth continuous flow might miss.

Summary of Intuition

The core takeaway of this research direction is that discretization is not a neutral act.

  1. The Step Matters: The finite size of $\eta$ creates a "drift" away from the loss you intended to minimize.
  2. Implicit Regularization: This drift acts as a hidden penalty term (usually on the gradient norm), favoring flatter regions of the parameter space.
  3. Optimizer Identity: Different update rules (Adam vs. SGD vs. Momentum) create different modified losses, leading to different generalization properties.

By treating the optimizer as a dynamical system and using Backward Error Analysis to find its "true" continuous objective, we move from trial-and-error hyperparameter tuning toward a rigorous physics of deep learning.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_backward_error_analysis_for_neural_optim_fe70b99a,
  title = {Backward Error Analysis for Neural Optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/backward-error-analysis-for-neural-optimization/rdr_fe70b99afcd5479b89baa425521cd327},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_backward_error_analysis_for_neural_optim_fe70b99a_v1,
  title = {Backward Error Analysis for Neural Optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/backward-error-analysis-for-neural-optimization/rdr_fe70b99afcd5479b89baa425521cd327/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}