In modern deep learning, normalization layers such as Batch Normalization, Layer Normalization, and Weight Normalization are ubiquitous. These components introduce a mathematical property known as scale invariance: if you multiply a layer's weights $w$ by a constant scalar $c$, the output of the layer remains completely unchanged ($f(cw) = f(x)$).
If the absolute scale of the weights does not alter the network's predictions, one might assume that weight magnitudes are irrelevant to training. However, the opposite is true. While weight scale does not change what the network computes, it profoundly dictates how the network learns by modulating the effective learning rate (ELR).
The Geometry of Scale Invariance
To understand why weight magnitude matters, we must look at the geometry of a gradient update. When an optimizer updates a weight vector via $w_{t+1} = w_t - \eta \nabla L(w_t)$, the actual functional change in a scale-invariant network depends entirely on the angular change—the shift in the weight vector's direction—rather than its length.
As observed by Tim Salimans et al. in Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks, the norm of unnormalized weight vectors naturally tends to grow during standard training. Sanjeev Arora et al. formalized the consequences of this growth in Theoretical Analysis of Auto Rate-Tuning by Batch Normalization. They proved that the effective learning rate—the step size controlling directional updates on the unit sphere—is inversely proportional to the squared norm of the weights:
$$\text{ELR} \propto \frac{\eta}{\|w\|^2}$$
Think of this using a steering wheel analogy. If the steering wheel's radius ($\|w\|$) is tiny, a one-inch physical push ($\eta \nabla L$) turns the car sharply. If the radius grows massive, that exact same one-inch push barely changes the car's direction.
Consequently, if parameter norms grow unbounded, the ratio of the gradient norm to the parameter norm decreases. This causes the effective learning rate to vanish, causing optimization to stagnate and leading to a loss of network plasticity, as highlighted by Daniel Palenicek et al. in Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalization.
Weight Decay as a Centripetal Force
In traditional neural networks, weight decay (L2 regularization) acts to restrict model capacity and prevent overfitting by penalizing large weights. In scale-invariant architectures, however, weight decay loses its role as a traditional regularizer because changing $\|w\|$ does not alter the underlying function. Instead, it functions purely as a mechanism to control the effective learning rate.
Daniel Kunin et al. mapped this interaction in Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics using a physical framework. A finite learning rate creates a centrifugal effect that pushes parameter norms outward, while weight decay acts as a centripetal force pulling them back inward.
The interaction between these two opposing forces establishes a dynamic equilibrium (or rotational equilibrium). As demonstrated by Ruosi Wan et al. in Spherical Motion Dynamics: Learning Dynamics of Normalized Neural Network using SGD and Weight Decay and Atli Kosson et al. in Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks, the weight norm stabilizes at a steady state where the outward expansion of stochastic gradient updates is perfectly countered by the contractive pull of weight decay. This equilibrium ensures that the angular update speed remains constant throughout training.
This perspective reveals a surprising equivalence. Zhiyuan Li et al. showed in An Exponential Learning Rate Schedule for Deep Learning that training a scale-invariant network with weight decay is mathematically identical to training it without weight decay but using an exponentially increasing global learning rate schedule. Weight decay's primary function is simply to shrink weight norms so that the effective step size remains high enough for continued exploration. Furthermore, Taiki Miyagawa noted in Toward Equation of Motion for Deep Neural Networks: Continuous-time Gradient Descent and Discretization Error Analysis that discrete step sizes introduce discretization errors that modify this effective decay rate, meaning continuous-time models must explicitly account for the learning rate to accurately predict equilibrium states.
Auto Rate-Tuning and Layer-Wise Balance
In deep architectures, different layers accumulate gradients at different rates. Why don't their effective learning rates diverge wildly, destabilizing the network?
The system is protected by a self-balancing negative feedback loop known as auto rate-tuning. As explored by Christian H. X. Ali Mehmeti-Göpel et al. in On the Weight Dynamics of Deep Normalized Networks, when a normalized layer experiences large gradient updates due to a high local ELR, its weight norm rapidly increases. Because the ELR is inversely proportional to the weight norm, this growth immediately forces the layer's effective learning rate to decay.
High Effective LR ➔ Large Gradient Updates ➔ Weight Norm Grows ➔ Effective LR Decreases
The authors of Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimes demonstrated that this negative feedback mechanism causes the effective step sizes of individual layers to automatically converge toward a shared equilibrium value. If a layer falls behind or rushes ahead, the internal weight dynamics naturally correct the step size, ensuring balanced optimization across the entire network architecture without manual layer-by-layer tuning.
The Edge of Stability and Periodic Spikes
While this equilibrium is robust, extreme hyperparameter choices can push the system into chaotic regimes.
Consider what happens when weight decay continuously shrinks the weight norm. As the norm drops, the effective learning rate climbs. If the weight norm becomes excessively small, the ELR explodes, pushing the optimizer past the "edge of stability."
Ekaterina Lobacheva et al. documented this phenomenon in On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decay. They showed that the interplay between scale invariance and weight decay can trigger a periodic training cycle:
- Contraction Phase: Weight decay steadily reduces the weight norm, causing the ELR to climb.
- Explosion Phase: The weight norm becomes critically small, causing the ELR to explode and triggering a sudden spike in training loss.
- Reset Phase: The massive gradients from the loss spike push the weight norm outward, lowering the ELR back to a stable regime and restarting the cycle.
Conversely, if weight decay is completely removed, the network suffers from severe underfitting because uncontrolled weight growth causes the effective step size to vanish, stalling optimization entirely (Xiang Li et al., Understanding the Disharmony between Weight Normalization Family and Weight Decay).
Managing this trade-off is central to generalization. By utilizing weight decay to maintain small weight norms, practitioners keep the intrinsic learning rate and gradient noise high. As shown by Kaifeng Lyu et al. in Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction, this dynamic forces the optimization process to behave like projected gradient descent on a hypersphere, pushing the model out of sharp, unstable regions and forcing it to converge toward flatter minima that generalize significantly better to unseen data (Johan Björck et al., Understanding Decoupled and Early Weight Decay).