In traditional optimization theory, we often treat the "stochastic" part of Stochastic Gradient Descent (SGD) as a simple additive error—like a static cloud of white noise following the optimizer. However, research into deep learning dynamics suggests a more complex reality: the noise is state-dependent (or endogenous). The noise doesn't just happen to the model; it is generated by the model's current position in the loss landscape.
The Myth of Constant Noise
Most classical proofs of convergence assume that gradient noise is independent and identically distributed (i.i.d.) with a constant variance. As Arnulf Jentzen and Benno Kuckuck point out in Strong error analysis for stochastic gradient descent optimization algorithms, this assumption fails even in simple linear regression. In practice, as the parameters grow or move, the variance of the noise scales accordingly.
This state-dependency creates a feedback loop. Jingzhao Zhang, Hongzhou Lin, and Suvrit Sra argue in Beyond Worst-Case Analysis in Stochastic Approximation that treating noise as an external, exogenous factor prevents us from understanding why adaptive optimizers like Adam outperform vanilla SGD. These algorithms succeed precisely because they estimate the "moments" of this state-dependent noise, adapting to the local geometry of the landscape.
Multiplicative Noise and the Power Law Escape
One of the most critical realizations in this field is that SGD noise is often multiplicative. Instead of being a fixed $\epsilon$, the noise magnitude is often proportional to the loss itself or the distance from the optimum. Liam Hodgkinson and Michael W. Mahoney demonstrate in Multiplicative Noise and Heavy Tails in Stochastic Optimization that this structure is what allows parameters to form heavy-tailed distributions, enabling the optimizer to "hop" between distant basins.
This changes the "physics" of how models escape local minima. In classical thermodynamics, escaping a potential well follows Kramers' Law, which predicts an exponential escape rate. But Takashi Mori and Ziyin Liu show in Power-Law Escape Rate of SGD that because SGD noise scales with the loss, the escape rate is actually a power law. This explains why SGD is so much more effective at finding flat minima than continuous-time models with constant noise would suggest: the noise "shuts off" in flat regions but remains violent in sharp ones.
Heavy Tails as a Feature, Not a Bug
If the noise were purely Gaussian, the probability of a large jump would be infinitesimally small. However, Umut Simsekli and Levent Sagun found that stochastic gradient noise is often heavy-tailed ($\alpha$-stable). In A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks, they argue that this non-Gaussianity is a direct result of the state-dependent nature of the data-generating process.
These heavy tails act as a natural regularizer. Zhe Jiao and Martin Keller-Ressel describe this in Emergence of heavy tails in homogenized stochastic gradient descent as a "homogenization" process where the discrete jumps of SGD create a complex equilibrium that favors generalizable solutions.
Theoretical Bridges: SDEs and Markovian Noise
To study these effects rigorously, researchers use Stochastic Differential Equations (SDEs) to model the discrete steps of SGD as a continuous flow. Aleksei Ustimenko and Aleksandr Beznosikov provide a unified framework for this in Ito Diffusion Approximation of Universal Ito Chains, proving that even with non-Gaussian, state-dependent noise, we can still derive tight convergence bounds.
Furthermore, in settings like Reinforcement Learning, the noise is not just state-dependent but Markovian—the noise at step $t+1$ depends on the state at step $t$. Prasenjit Karmakar has developed foundational theory for Two Timescale Stochastic Approximation with Controlled Markov noise, showing how algorithms can converge even when the noise is biased by the algorithm's own previous decisions.
Why It Works: The Intuition
The reason state-dependent noise is so powerful is that it makes the optimizer self-aware.
- In sharp regions: The gradient is large and the noise is high, forcing the optimizer to move away quickly.
- In flat regions: The noise magnitude drops, allowing the optimizer to settle and "explore" the local area.
- At the optimum: In overparameterized models, the noise can vanish entirely (interpolation), allowing for exact convergence as shown by Adrien Schertzer in Stochastic Differential Equations models for Least-Squares SGD.
By moving away from the "additive noise" simplification, we see that the "noise" in SGD is not a hindrance to be minimized, but a sophisticated mechanism that navigates the complex geometry of deep learning.