Fluctuation-dissipation relations for stochastic gradient descent

Sho YaidaICLR 2019

Introduction

In deep learning, setting the "learning rate schedule"—deciding exactly when to decrease the step size—is often more of an art than a science. Practitioners usually rely on trial and error or preset milestones (e.g., "drop the learning rate at epoch 100").

Theoretically, many researchers treat Stochastic Gradient Descent (SGD) as a continuous-time Stochastic Differential Equation (SDE). However, this paper argues that the discrete nature of SGD steps actually matters. There is a lack of principled, measurable metrics to determine if a model has reached a "stationary state" (a plateau where it is just bouncing around a local minimum) and to understand the geometric "shape" of the loss landscape the model is currently exploring. This paper bridges statistical mechanics and deep learning by deriving "fluctuation-dissipation relations" (FDRs) specifically for SGD to solve these issues.

Methods: Deriving FDRs for SGD

The author derives two primary stationary fluctuation-dissipation relations based on the "Master Equation" of the discrete SGD process:

  1. FDR1 (The Stationarity Metric): This relation links the alignment of the weights with the gradient to the learning rate and the noise matrix. It provides a concrete number that should equal 1 when the model has reached equilibrium.
  2. FDR2 (The Landscape Probe): This relation links the magnitude of the gradient to the Hessian (curvature) of the loss function and "anharmonicity" (how much the landscape deviates from a simple bowl shape).

Crucially, these derivations do not assume that the noise is Gaussian or that the loss function is convex, making them applicable to real-world deep neural networks. Based on FDR1, the author proposes an adaptive learning rate scheduler that monitors the stationarity metric and automatically reduces the learning rate when the model plateaus.

Stationarity metric converging over epochs for MLP on MNIST and CNN on CIFAR-10.

Key Insights

  • SGD $\neq$ SDE: The paper highlights a subtle but important technical point: treating SGD as a continuous SDE is often inconsistent because the noise scaling in discrete SGD doesn't match the square-root scaling required by standard calculus (Ito/Stratonovich). The discrete-time approach used here is more physically sound.
  • Measuring "Plateaus": FDR1 provides a "thermometer" for training. By calculating the ratio between the work done by the gradient and the noise injected by mini-batches, we can tell exactly when the model has stopped making progress.
  • Anharmonicity Matters: The paper shows that for complex models like CNNs on CIFAR-10, the loss landscape is highly "anharmonic" (complex and non-quadratic) even at small learning rates. This invalidates simpler theories that assume the loss looks like a perfect parabolic bowl.
  • Principled Scheduling: Instead of guessing when to drop the learning rate, we can wait for the FDR1 metric to saturate.

Empirical Results

The theoretical claims were empirically verified using two standard benchmarks: a multilayer perceptron (MLP) trained on MNIST using SGD without momentum, and a Convolutional Neural Network (CNN) trained on CIFAR-10 using SGD with momentum.

Validation of fluctuation-dissipation relations across different learning rates for both MLP and CNN architectures.
  • Validation of FDRs: In both MLP and CNN experiments, the theoretical relations (FDR1 and FDR2) held true once the models reached stationarity across various learning rates.
  • Adaptive Success: The proposed adaptive scheduler (which drops the learning rate based on the FDR1 metric) achieved comparable or better test accuracy than preset schedules.
  • Outperforming AMSGrad: On the MNIST task, the adaptive SGD method reached higher accuracy and converged more reliably than the popular AMSGrad optimizer.
  • Landscape Insights: The author used FDR2 to show that CNNs experience much higher degrees of landscape complexity (anharmonicity) than simpler MLPs, providing a tool to "see" the geometry of the loss function during training.

Limitations and Future Work

While promising, the approach has a few limitations. First, the relations rely on the model eventually reaching a stationary distribution. In cases like unregularized cross-entropy loss, weights can sometimes "cascade" toward infinity, potentially breaking the assumption of stationarity. Second, while FDR1 is easy to compute on the fly, FDR2 involves terms that are more expensive to calculate (like the full-batch gradient), though sparse sampling is suggested as a workaround. Finally, the experiments were limited to relatively simple models (MLP and small CNNs). Future work is needed to see how these relations behave in massive architectures like Transformers or in complex Natural Language Processing tasks.

Summary updated

BibTeX
@inproceedings{yaida2019fluctuation,
  author = {Sho Yaida},
  title = {{Fluctuation-dissipation relations for stochastic gradient descent}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
}
Figure