Keep the Momentum: Conservation Laws beyond Euclidean Gradient Flows

Sibylle Marcotte, Rémi Gribonval, Gabriel Peyré

Introduction

In the study of neural network optimization, understanding the dynamical path of weights is as crucial as the final loss value. Standard Euclidean gradient descent is known to obey certain "conservation laws"—mathematical invariants like the balancedness of weight magnitudes between layers. These laws provide theoretical grounding for convergence and generalization. However, modern training often employs momentum-based methods and non-Euclidean metrics. This paper investigates whether these vital conservation laws persist in these more complex, practical settings.

Theoretical Framework: Optimization as Physics

To analyze these dynamics, the researchers treat optimization as a physical system. By employing Phase-Space Lifting, second-order momentum equations are transformed into a first-order system of position (weights) and velocity. The study utilizes a Lie Algebra framework to analyze the symmetries of the vector fields defining the training path.

Central to this approach is Noether’s Theorem, which links physical symmetries to conservation laws. By applying this to the Lagrangian formulations of optimization flows, the authors developed an algorithmic procedure to identify all possible polynomial conservation laws within a given system.

The Fragility of Conservation

The investigation reveals that conservation laws are surprisingly fragile. The introduction of momentum often acts as a "law-breaker," particularly in non-linear architectures.

The Role of Momentum

In standard gradient descent, conservation laws are static functions of the weights. With momentum, these laws transform into dynamic entities that depend on both time and the velocity of weight updates. For two-layer ReLU networks, the study proves that every conservation law present in gradient flow vanishes entirely upon the addition of momentum.

Geometry and Over-parameterization

The choice of metric—the geometry of the weight space—is equally critical. While Natural Gradient Flow tends to preserve standard laws, mirror descent geometries (often used in Non-negative Matrix Factorization) lose their invariants under momentum. Interestingly, linear networks exhibit a "paradox": while mildly over-parameterized models lose laws, highly over-parameterized regimes can actually generate more conservation laws than their gradient descent counterparts.

Key Results

The theoretical predictions were validated through simulations on MNIST and synthetic models. The researchers quantified the exact number of conservation laws for various settings, revealing a consistent trend of "conservation loss" when moving from gradient flow to momentum.

Summary of conservation law counts across different architectures and geometries. The blue values represent laws in gradient flow, while red values represent those in momentum flow.

Key findings include:

  • ReLU Networks (MLP): Momentum reduces the number of conservation laws from $r$ to zero, indicating that the balancedness properties of ReLU networks are entirely lost under momentum.
  • Linear Networks (PCA): There is a precise loss of $r$ laws when the hidden dimension $r$ is less than the sum of input and output dimensions ($n+m$).
  • NMF & ICNN: While new laws were identified for gradient flows in these non-Euclidean settings (Theorems 4.5 and 4.6), they do not persist when momentum is introduced.

Conclusion and Future Directions

While this work provides a comprehensive characterization for continuous-time flows, several questions remain. Numerical experiments suggest these laws are approximately preserved in discrete-time algorithms, but a formal discrete-time theory is yet to be established. Additionally, the impact of stochasticity in algorithms like SGD presents a significant frontier for future research into training stability and the "boundedness" of these quantities.

Summary updated

BibTeX
@inproceedings{marcotte2024keep,
  author = {Sibylle Marcotte and Rémi Gribonval and Gabriel Peyré},
  title = {{Keep the Momentum: Conservation Laws beyond Euclidean Gradient Flows}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=hG6gddAKnJ},
}
Figure