Noether's Learning Dynamics: The Role of Kinetic Symmetry Breaking in Deep Learning

Hidenori Tanaka, Daniel Kunin

Abstract

This work bridges classical physics and deep learning by modeling neural network training as a physical system governed by Lagrangian mechanics. By introducing the concept of Kinetic Symmetry Breaking (KSB)—where a learning rule breaks the symmetries of a loss function—the authors provide a mathematical explanation for the effectiveness of normalization layers. This framework, termed Noether’s Learning Dynamics (NLD), reveals that Batch Normalization acts as an implicit adaptive optimizer, functionally equivalent to RMSProp.

Introduction: The Mystery of Symmetry in Training

In physics, symmetry is a foundational principle: rotating a circular bowl does not change its shape. Deep learning architectures often incorporate similar symmetries; for example, Batch Normalization (BatchNorm) ensures the loss function is "scale-invariant," meaning the loss remains constant even if weights are scaled.

However, a significant gap exists in optimization theory: while the loss landscape (potential energy) may be symmetric, the training trajectory (the path weights take) often is not. Standard theories struggle to unify the static geometry of the loss landscape with the dynamic behavior of learning rules. This paper introduces Noether’s Learning Dynamics (NLD) to understand how the interaction between architecture and learning rules determines training success.

Methods: Training as a Physical System

The authors treat the discrete steps of gradient descent as a continuous-time physical system. By accounting for finite learning rates, they introduce an "implicit acceleration" term, transforming first-order gradient flow into a second-order system akin to a particle with mass.

Analogy between learning dynamics in parameter space and classical mechanics in physical space.

Central to their method is the definition of Kinetic Symmetry Breaking (KSB). KSB occurs when the "kinetic energy" (the learning rule) explicitly breaks the symmetries of the "potential energy" (the loss function). While BatchNorm makes the loss scale-invariant, standard SGD is not scale-invariant, leading to KSB. The authors generalize Noether’s Theorem—which links symmetries to conservation laws—to account for the damping and symmetry-breaking inherent in machine learning.

Key Insights: The "Aha!" Moment on BatchNorm

The most striking insight is that BatchNorm does more than just "smooth" the loss landscape; it mathematically implements a form of implicit adaptive optimization. The authors prove that the way BatchNorm scales gradients over time is functionally identical to the RMSProp optimizer.

Visualizing symmetry breaking: (a) rotation symmetry in a physical system where dynamics remain invariant, and (b) scale symmetry in a neural network with BatchNorm where the learning dynamics diverge, demonstrating Kinetic Symmetry Breaking.

In this context, symmetry breaking is a feature, not a bug. Breaking scale symmetry allows the network to automatically tune its own effective learning rate, leading to faster and more stable convergence. Furthermore, momentum acts as an amplifier, increasing this adaptive optimization effect by roughly 1900 times in standard configurations ($\beta=0.9$).

Empirical Results and Validation

The authors validated their theory using VGG11 models on the Tiny-ImageNet dataset (100,000 images across 200 classes). Key findings include:

  • Mathematical Equivalence: The derived exact solution for the effective learning rate matches the update rule of RMSProp.
  • Performance Gap: "Unconstrained" models (permitting KSB) significantly outperformed "constrained" models where symmetry breaking was artificially suppressed. This gap was most pronounced at higher learning rates and momentum.
  • Weight Decay: In normalized networks, weight decay plays a role identical to the "discount factor" ($\rho$) in RMSProp, controlling the memory of past gradients.

Limitations and Future Work

The theory currently relies on a continuous-time approximation of gradient descent, which may not capture all discrete-time "jitter." While the focus was on scale, rotation, and translation symmetries, future work could extend NLD to complex permutation symmetries in deep ReLU networks. Additionally, the authors suggest that directly simulating gradient flow with implicit acceleration could allow researchers to reap the benefits of large learning rates without the instability of standard discrete updates.

Summary updated

BibTeX
@inproceedings{tanaka2021noether,
  author = {Hidenori Tanaka and Daniel Kunin},
  title = {{Noether's Learning Dynamics: The Role of Kinetic Symmetry Breaking in Deep Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=fiPtD7iXuhn},
}
Figure