Effective Learning Rate Dynamics in Scale-Invariant Networks

This direction is merged into another direction: effective learning rate decay. Its last version stays readable and citable here.

Version 1 of 113 papers26 conceptsfrozen 2026-10-04

In modern deep learning, layers like Batch Normalization and Layer Normalization have become ubiquitous. These layers introduce a property called scale invariance: if you multiply the weights of a preceding layer by a constant factor, the normalization layer simply divides them back out, leaving the network's output unchanged.

While this stabilizes training, it creates a strange geometric paradox. Because the model's behavior depends only on the direction of the weight vector and not its magnitude, the traditional intuition for optimization breaks down. A research direction involving authors like Byeongho Heo, Clare Lyle, and Yucong Zhou has revealed that in these networks, the norm of the weights acts as a hidden "throttle" on the learning process, leading to an unintended decay of the effective learning rate.

The Geometry of the Effective Learning Rate

To understand why weight magnitude matters in a scale-invariant system, imagine a weight vector $w$ on a sphere. Since the scale doesn't matter, learning is essentially the process of moving $w$ across the surface of that sphere to find the best direction.

If the weight vector is short (small norm), a single gradient update step $\Delta w$ will significantly change its direction. However, if the weight vector is very long (large norm), that same step $\Delta w$ will barely nudge the direction at all. Mathematically, the "effective" step size is inversely proportional to the norm of the weights ($\|w\|$).

As Byeongho Heo and colleagues explain in AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights, this means that as weights grow, the optimizer's progress naturally slows down. This is the Effective Learning Rate (ELR): the actual speed at which the model's representational logic evolves, regardless of the nominal learning rate set in the code.

The Weight Norm Growth Trap

The problem is that standard optimizers, particularly those using momentum like Adam or SGD, tend to push weight norms to grow continuously. In AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights, the authors demonstrate that momentum-based updates accumulate historical gradient information into the radial component of the weight vector, causing the norm to grow much faster than in vanilla gradient descent—sometimes up to 19 times faster.

This rapid growth triggers a "stealthy" decay of the ELR. If the weights grow too large, the ELR can drop toward zero, causing the model to stop adapting entirely. In the context of Reinforcement Learning, Clare Lyle and Hado Philip van Hasselt identify this as a primary cause for the "loss of plasticity," where agents become unable to learn new information in non-stationary environments because their weight norms have grown too large to allow for meaningful updates (Normalization and effective learning rates in reinforcement learning).

The Hidden Role of Weight Decay

In non-normalized networks, weight decay is usually viewed as a way to limit model complexity (regularization). However, in scale-invariant networks, its role changes completely.

As Yucong Zhou and Yunxiao Sun argue in FixNorm: Dissecting Weight Decay for Training Deep Neural Networks, weight decay in these architectures primarily serves to shrink the weight norm, which counter-intuitively increases the effective learning rate. By pulling the weights back toward the origin, weight decay prevents the ELR from decaying too far.

This creates a delicate "Spherical Motion Dynamics" (SMD). Ruosi Wan and Zhanxing Zhu show that training reaches an equilibrium where the outward pressure of gradient updates is balanced by the inward pull of weight decay (Spherical Motion Dynamics of Deep Neural Networks with Batch Normalization and Weight Decay). When you manually drop the learning rate during training, this equilibrium is shattered, and the network must spend many epochs slowly adjusting its weight norm to a new stable state, leading to inefficient "lag" periods in the training curve.

Strategies for Stabilization

Recognizing that weight norm growth is the root cause of ELR instability, researchers have proposed several mechanisms to decouple weight magnitude from learning speed:

  1. Projection (AdamP/SGDP): Byeongho Heo proposed projecting out the norm-increasing component of the gradient update. By ensuring updates are strictly tangential to the weight vector, the norm stays constant, and the ELR remains stable.
  2. Explicit Rescaling (WRS/FixNorm): Instead of relying on the indirect tug-of-war of weight decay, methods like Weight Rescaling (WRS) by Ziquan Liu and Antoni B. Chan explicitly constrain weights to a unit norm. This eliminates the hyperparameter sensitivity of weight decay and prevents late-stage overfitting (Weight Rescaling: Effective and Robust Regularization for Deep Neural Networks with Batch Normalization).
  3. Weight Normalization in RL: In high-throughput reinforcement learning, Daniel Palenicek and Jan Peters found that combining Batch Normalization with Weight Normalization prevents the "stealthy decay" of step sizes, allowing agents to learn efficiently from much less data without needing to reset the network (Scaling CrossQ with Weight Normalization).

By shifting the focus from the nominal learning rate to the effective learning rate, this research direction provides a clearer lens for understanding why certain architectures stall and how to design optimizers that maintain their plasticity throughout the entire training process.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_effective_learning_rate_dynamics_in_scal_3139ce8f,
  title = {Effective Learning Rate Dynamics in Scale-Invariant Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/effective-learning-rate-dynamics-in-scale-invariant-networks/rdr_3139ce8f7bc04eb0b5a04f7fd18bba15},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_effective_learning_rate_dynamics_in_scal_3139ce8f_v1,
  title = {Effective Learning Rate Dynamics in Scale-Invariant Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/effective-learning-rate-dynamics-in-scale-invariant-networks/rdr_3139ce8f7bc04eb0b5a04f7fd18bba15/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}