Noether’s Learning Dynamics: Role of Symmetry Breaking in Neural Networks

Hidenori Tanaka, Daniel Kunin

This paper provides a fascinating bridge between the centuries-old principles of classical mechanics and the modern "black box" of deep learning. By treating the training of a neural network as the motion of a physical particle, the authors uncover a hidden geometric mechanism that explains why modern training tricks—like Batch Normalization and Weight Decay—actually work.

Analogy between learning dynamics and classical mechanics.

TL;DR

The authors develop a theoretical framework called Noether’s Learning Dynamics (NLD) by applying Lagrangian mechanics to neural network training. They reveal that when a learning rule "breaks" the symmetry of a loss function (a phenomenon they call Kinetic Symmetry Breaking), it creates an implicit adaptive optimization effect mathematically equivalent to RMSProp.

The Role of Symmetry and Symmetry Breaking

In deep learning, we often talk about the "loss landscape"—the high-dimensional hills and valleys the model navigates during training. We know that certain architectures possess inherent symmetries. For example, in a network with Batch Normalization (BN), doubling the weights of a layer doesn't change the output because the normalization step cancels the scaling out. This is known as "scale symmetry."

However, even if the loss landscape is symmetric, the path the model takes during training is not. If you start with larger weights, the gradient updates behave differently than if you start with smaller ones.

Scale symmetry and the effect of initialization on loss.

There is a gap in our understanding: how do the static geometry of the architecture and the dynamic rules of the optimizer (like SGD with momentum) interact? Specifically, why does breaking these symmetries often lead to better, more stable training?

A Physics-Based Methodology

The authors bridge this gap using Lagrangian Mechanics, translating physical concepts into machine learning phenomena:

  1. The Lagrangian of Learning: They model discrete Gradient Descent as a continuous-time system. In this analogy, the Loss Function acts as the Potential Energy, and the Learning Rule (how we update parameters) corresponds to the Kinetic Energy.
  2. Implicit Acceleration: Using a finite learning rate introduces an "implicit acceleration" term, turning the first-order gradient flow into a second-order system—much like a particle with mass and momentum.
  3. Kinetic Symmetry Breaking (KSB): This is the paper's core contribution. KSB occurs when the loss function has a symmetry (like scaling), but the kinetic energy (the optimizer's behavior) does not respect it.
  4. Noether’s Theorem Generalization: In physics, Noether’s theorem states that every symmetry leads to a conserved quantity (like momentum). The authors generalize this to account for KSB, deriving an equation for how these "conserved quantities" (Noether charges) actually evolve when the symmetry is broken.

Key Insights: The RMSProp Connection

Usually, physicists look for symmetries to simplify systems. Here, the authors show that breaking symmetry is actually a feature that allows for "auto-rate tuning." Because the optimizer doesn't respect the scale symmetry of BatchNorm, it effectively adjusts its own effective learning rate.

This leads to a stunning mathematical equivalence. The authors show that the dynamics induced by BatchNorm + Weight Decay + Momentum are functionally identical to the update rule of RMSProp.

  • Weight Decay as a Discount Factor: In RMSProp, a hyperparameter discounts old gradient information. The authors show that in standard SGD training, Weight Decay plays exactly this role.
  • Momentum as an Amplifier: Momentum doesn't just speed up training; it exponentially amplifies the adaptive learning effect. In a standard setup, momentum can amplify this adaptation by a factor of nearly 1900x.

Empirical Validation and Future Work

The authors successfully derived the NLD equation, which predicts how the norm of the weights changes over time based on the history of gradients. By training VGG11 on Tiny-ImageNet, they showed that "unconstrained" models (which allow the weight norms to evolve and break symmetry) significantly outperform "constrained" models (where weight norms are frozen). Their formula for the "effective learning rate" accurately matched the actual behavior of neural networks during training.

Limitations and Next Steps: * Continuous vs. Discrete: The theory relies on a continuous-time approximation of gradient descent. Real-world training happens in discrete steps, which can introduce noise not fully captured here. * Symmetry Types: The paper focuses on continuous differentiable symmetries (scale, rotation, translation). It does not yet account for discrete symmetries, such as permuting the order of neurons. * Activation Functions: Extending this framework to the "rescale symmetry" found in ReLU networks (without normalization) is a promising next step.

By framing deep learning through the lens of Noether and Lagrange, this paper moves the field away from trial-and-error hyperparameter tuning and toward a fundamental geometric theory of why neural networks learn the way they do.

Summary updated

Abstract

In nature, symmetry governs regularities, while symmetry breaking brings texture. In artificial neural networks, symmetry has been a central design principle to efficiently capture regularities in the world, but the role of symmetry breaking is not well understood. Here, we develop a theoretical framework to study the "geometry of learning dynamics" in neural networks, and reveal a key mechanism of explicit symmetry breaking behind the efficiency and stability of modern neural networks. To build this understanding, we model the discrete learning dynamics of gradient descent using a continuous-time Lagrangian formulation, in which the learning rule corresponds to the kinetic energy and the loss function corresponds to the potential energy. Then, we identify "kinetic symmetry breaking" (KSB), the condition when the kinetic energy explicitly breaks the symmetry of the potential function. We generalize Noether’s theorem known in physics to take into account KSB and derive the resulting motion of the Noether charge: "Noether's Learning Dynamics" (NLD). Finally, we apply NLD to neural networks with normalization layers and reveal how KSB introduces a mechanism of implicit adaptive optimization, establishing an analogy between learning dynamics induced by normalization layers and RMSProp. Overall, through the lens of Lagrangian mechanics, we have established a theoretical foundation to discover geometric design principles for the learning dynamics of neural networks.
BibTeX
@inproceedings{tanaka2021noethers,
  author = {Hidenori Tanaka and Daniel Kunin},
  title = {{Noether’s Learning Dynamics: Role of Symmetry Breaking in Neural Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=fiPtD7iXuhn},
}
Figure