In deep learning, we often visualize the "loss landscape"—a high-dimensional map where the elevation represents the model's error. A long-standing intuition in the field is that "flat" minima (broad valleys) generalize better to new data than "sharp" minima (narrow pits). However, this geometric intuition is complicated by a fundamental mathematical property of modern neural networks: weight scale invariance.
The Rescaling Symmetry
Most modern architectures rely on Rectified Linear Units (ReLU) or normalization layers (like Batch Norm). These components introduce a specific symmetry: you can multiply the weights of one layer by a constant $\alpha$ and divide the weights of the next layer by the same $\alpha$ without changing the network's output.
As Laurent Dinh et al. demonstrated in Sharp Minima Can Generalize For Deep Nets, this "positive scale invariance" means that a single mathematical function can be represented by infinitely many points in the parameter space. Because ReLU is positively homogeneous—meaning $ReLU(\alpha x) = \alpha ReLU(x)$ for $\alpha > 0$—the internal activations can be scaled up or down across layers while the final prediction remains identical.
The Sharpness Illusion
The existence of these symmetries creates a paradox for our understanding of generalization. If we can transform a "flat" minimum into an "arbitrarily sharp" one just by rescaling the weights, then traditional measures of sharpness—like the Hessian (the second derivative of the loss)—cannot be intrinsic properties of the model's performance.
W. Ronny Huang et al. argue in Understanding Generalization Through Visualizations that this sensitivity to weight rescaling makes it difficult to link the geometry of the loss landscape to model performance. If a simple coordinate transformation can manipulate the perceived flatness of a minimum without changing the model's output, then Euclidean-based metrics are fundamentally unreliable.
Constructing Invariant Metrics
To solve this, researchers have developed metrics that remain constant regardless of how weights are scaled. The goal is to measure the "intrinsic" sharpness of the function rather than the "extrinsic" sharpness of the specific weight values.
- Magnitude-Aware Perturbations: Instead of checking how the loss changes when weights are shifted by a fixed amount, Jungmin Kwon and Jeongseop Kim proposed ASAM in ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks. ASAM adaptively scales the perturbation region based on the weight magnitudes, ensuring the optimizer searches for minima that are truly flat in a scale-invariant sense.
- Information Geometry: Cheongjae Jang et al. introduced Information Geometric Sharpness (IGS) in A Reparametrization-Invariant Sharpness Measure Based on Information Geometry. By measuring sharpness relative to the change in the model's output distribution (using the Fisher Information Matrix) rather than the change in weight values, the metric becomes immune to reparameterization.
- Normalization-Specific Metrics: For networks using Batch Normalization, Mingyang Yi et al. developed BN-Sharpness in BN-invariant Sharpness Regularizes the Training Model to Better Generalization, which accounts for the fact that BN layers project weights onto a unit sphere, making their absolute norm irrelevant to the function.
Rethinking Regularization and Dynamics
Scale invariance doesn't just affect how we measure models; it changes how they learn. Standard L2 regularization (weight decay) is designed to penalize large weights, but in a scale-invariant network, the model can "hide" complexity by shifting weight scales between layers to minimize the penalty without changing the function.
Yufei CUI et al. addressed this in Improve Generalization and Robustness of Neural Networks via Weight Scale Shifting Invariant Regularizations by proposing WEISSI, a regularizer that is invariant to these shifts. Similarly, Wei Chen et al. proposed Path-BN in Path-BN: Towards effective batch normalization in the Path Space for ReLU networks, arguing that optimization should happen in "path space"—a representation that is naturally invariant to the scaling of individual weights.
From a theoretical perspective, these symmetries imply the existence of "conservation laws" in training. Hidenori Tanaka and Daniel Kunin showed in Noether’s Learning Dynamics that the rescaling symmetry in neural networks leads to invariant quantities that remain constant during SGD, much like energy or momentum are conserved in physics. Understanding these invariants allows us to explain why certain training tricks, like the combination of weight decay and normalization, are so effective at navigating the complex, redundant geometry of the loss landscape.
Key Foundational Work
Path-SGD: Path-Normalized Optimization in Deep Neural Networks (2015) — contributed 3 concept elements to this direction.