Abstract
Understanding the geometry of neural network loss functions is critical for improving optimization and generalization. However, high-dimensional non-convexity and weight scale invariance often make these landscapes difficult to visualize accurately. This paper introduces "filter-wise normalization," a technique that enables meaningful side-by-side comparisons of loss surfaces. We demonstrate that architectural choices, particularly skip connections, fundamentally smooth the loss landscape, preventing a transition to chaotic geometry as network depth increases.
Introduction
Neural networks are typically trained by minimizing a loss function in a high-dimensional parameter space. While empirical "tricks" like skip connections (ResNets) or specific batch sizes are known to improve performance, the underlying geometric reasons have remained obscure. Previous visualization techniques, such as 1D linear interpolation, often produced misleading results because neural networks are scale-invariant: weights can be rescaled across layers without changing the model's output. Consequently, a loss minimizer might appear "sharp" or "flat" simply due to the arbitrary scale of its weights rather than its intrinsic geometry.
Methods: Filter-Wise Normalization
To address the limitations of standard visualization, the authors propose Filter-Wise Normalization. To visualize a loss landscape in 2D, researchers typically select a center point (the trained weights) and two random direction vectors to define a plane.
Standard methods use random directions without considering weight scale. Li et al. argue that direction vectors should be normalized filter-by-filter. Specifically, if a filter in the model has a certain norm, the corresponding part of the random direction vector is scaled to match that norm. This ensures that the perturbation applied during visualization is proportional to the weights themselves, neutralizing the deceptive effects of scale invariance and providing a "fair" comparison between different models.
Key Insights: The Impact of Architecture
The most significant finding is the dramatic effect of skip connections on the loss landscape.
Skip Connections as Landscape Smoothers
Without skip connections, deep networks (e.g., ResNet-56 or ResNet-110) exhibit "chaotic" loss landscapes characterized by numerous local minima and jagged hills. This complexity makes optimization extremely difficult. Adding skip connections prevents this transition to chaos, maintaining a nearly convex, "bowl-like" landscape even at great depths.
The Chaos Threshold and Width
There is a clear "chaos threshold" as networks deepen. While a 20-layer "plain" network might appear relatively smooth, a 56-layer version becomes a jagged mess. Conversely, increasing the "width" of the network (the number of filters per layer) promotes flatter minimizers and wider regions of convexity, which correlates strongly with improved generalization.
Key Results and Generalization
The study provides visual and quantitative evidence that flatter landscapes correspond to better test performance. For instance, ResNet-56 without skip connections (chaotic landscape) had a test error of 13.31%, whereas the standard ResNet-56 (smooth landscape) achieved 5.89%.
Further analysis using Hessian eigenvalues validated these visual findings. Chaotic landscapes showed significant negative eigenvalues, indicating high non-convexity, while smooth landscapes had near-zero negative eigenvalues. Additionally, by projecting training trajectories using PCA, the authors found that 40% to 90% of SGD movement occurs within a very low-dimensional subspace, suggesting that the effective search space is much smaller than the total parameter count.
Conclusion
Filter-wise normalization provides a robust lens for studying the internal geometry of neural networks. By smoothing the loss landscape, skip connections allow for the training of much deeper networks than were previously possible. While 2D projections have limitations—they can prove the existence of non-convexity but not its absence—they offer invaluable intuition for designing more trainable and generalizable architectures.
Summary updated