An SDE perspective on stochastic inertial gradient dynamics with time-dependent viscosity and Hessian-driven damping

Rodrigo Maulen-Soto, Jalal Fadili, Hédy Attouch, Peter OchsOptimization 2025 · DOI

Optimization is the heartbeat of modern machine learning, but the algorithms we use—like Stochastic Gradient Descent (SGD)—are often noisy and prone to oscillations. To speed things up, researchers frequently turn to "inertial" methods (think Nesterov’s acceleration), which act like a ball rolling down a hill with momentum. However, momentum can lead to overshooting and wild "bouncing" around the minimum.

This paper, recently published in Optimization, provides a rigorous mathematical foundation for a sophisticated fix: combining time-dependent friction with damping driven by the function's curvature (the Hessian), all within a stochastic framework.

TL;DR

The authors develop a general mathematical framework using Stochastic Differential Equations (SDEs) to analyze fast, inertial gradient methods that use both time-varying friction and Hessian-driven damping. They prove that these systems achieve optimal convergence rates even when the gradient input is noisy, bridging the gap between continuous dynamical systems and practical stochastic optimization.

What Problem Does This Paper Address?

In the deterministic world, we know that adding "viscous damping" (friction that changes over time) can accelerate convergence. Adding "Hessian-driven damping" further helps by suppressing oscillations—essentially acting as a smart brake that engages when the terrain gets curvy.

However, most real-world optimization (like training a deep neural network) is stochastic. We don't have the true gradient; we have a noisy estimate.

  1. The Gap: Most existing analysis for these high-order damping systems is deterministic.
  2. The Challenge: Proving that these complex second-order systems still converge quickly—and don't just fly off into the weeds—when subjected to continuous random noise is mathematically daunting.

Methods

The authors model the optimization process as a Stochastic Differential Equation (SDE). Specifically, they study the Stochastic Inertial System with Implicit Hessian-driven Damping (S-ISIHD).

Key components of their approach include:

  • Stochastic Lyapunov Analysis: This is the "secret sauce." They construct complex energy functions (Lyapunov functions) and use Itô’s calculus to prove that these functions decrease over time, even with noise.
  • Implicit Hessian Damping: They favor an "implicit" formulation. Why? Because the "explicit" version would require taking the derivative of random noise (a martingale), which is mathematically undefined. The implicit version bypasses this, making it a better proxy for real-world algorithms.
  • Hamiltonian Reformulation: They transform the second-order SDE into a system of coupled first-order equations, allowing for a more flexible analysis of the long-term behavior of the trajectories.

Datasets

Not discussed in the paper. This is a theoretical work focused on mathematical proofs, convergence rates, and asymptotic behavior rather than empirical benchmarking on specific datasets.

Key Insights and Ideas

  • Noise Decay is Non-Negotiable: For the system to converge to the exact minimum (rather than just a neighborhood around it), the variance of the stochastic noise must vanish over time at a specific rate.
  • Preserving the "Fast" Rate: One of the most important findings is that the system can still achieve the "Nesterov-like" $O(1/t^2)$ convergence rate for convex functions in expectation, provided the noise integrates away fast enough.
  • Curvature as a Stabilizer: The Hessian-driven damping ($\beta$) acts as a geometric stabilizer. It allows the system to use higher momentum without the typical penalty of massive oscillations, even in the presence of noise.
  • Trajectory Convergence: Beyond just showing the function value gets small, the authors prove "almost sure weak convergence" of the actual path. This means the algorithm's iterates are guaranteed to settle down to a specific minimizer with probability 1.

Key Results

  • Convex Case: The authors established a convergence rate of $O(1/t^2)$ in expectation for the objective value, matching the best-known rates for deterministic accelerated methods.
  • Strongly Convex Case: They proved linear (exponential) convergence rates when the noise vanishes appropriately.
  • Almost Sure Bounds: They provided "almost sure" convergence results, which are much stronger than "in expectation" results because they guarantee the behavior of a single run of the algorithm, not just the average of many runs.

Limitations and Future Work

  • Smoothness Requirements: The current proofs assume the objective function is twice differentiable ($C^2$). Many machine learning problems involve non-smooth components (like ReLU activations or L1 penalties), which aren't covered here.
  • Discretization Gap: The paper focuses on the continuous-time SDE. While SDEs are excellent proxies, the "discretization error" introduced when turning these into actual computer code (discrete steps) remains an area for further refinement.
  • Parameter Tuning: While the paper gives the conditions for the damping parameters $\gamma(t)$ and $\beta(t)$, finding the optimal schedule for a specific practical task (like training a Transformer) is still an open engineering challenge.

Summary updated

Abstract

International audience
BibTeX
@inproceedings{maulensoto2025sde,
  author = {Rodrigo Maulen-Soto and Jalal Fadili and Hédy Attouch and Peter Ochs},
  title = {{An SDE perspective on stochastic inertial gradient dynamics with time-dependent viscosity and Hessian-driven damping}},
  booktitle = {Optimization},
  year = {2025},
  doi = {10.1080/02331934.2025.2526726},
}
Figure