Abstract
We examine the eigenvalues of the Hessian of a loss function before and after training. The eigenvalue distribution is composed of two distinct parts: the "bulk," which is concentrated around zero, and the "edges" or "spikes," which are scattered away from zero. We present empirical evidence indicating that the bulk reflects the degree of over-parametrization in the system, while the edges depend primarily on the input data.
Introduction
In deep learning, optimization is often conceptualized as navigating a high-dimensional loss landscape. While gradients provide the direction for descent, the second-order structure—the Hessian matrix—defines the landscape's local geometry, including its curvatures, valleys, and plateaus. Traditional optimization theory frequently assumes a non-singular Hessian, implying well-defined curvature in all directions. However, deep neural networks are typically over-parametrized, possessing far more parameters than training examples. This paper investigates the actual structure of the Hessian in such systems, revealing a remarkably "singular" reality that challenges standard assumptions.
Methodology
Analyzing the Hessian is computationally intensive, as its size grows quadratically with the number of parameters. To address this, the authors employed:
- Exact Hessian-Vector Products: Utilizing the Pearlmutter (1994) algorithm to compute products without explicitly storing the full matrix.
- Full Hessian Calculation: For smaller synthetic models, the full matrix was computed to visualize heatmaps and exact spectra.
- Systematic Variation: The researchers varied the number of hidden units and input data complexity (MNIST, synthetic 2D Gaussian blobs, and random patterns) to observe effects on the eigenvalue distribution.
The Structure of the Hessian: Bulk and Spikes
A central finding is that the Hessian's eigenvalues fall into two distinct categories, creating a "two-phase spectrum."
- The Bulk: A massive concentration of eigenvalues at or very near zero. This component is sensitive to the network architecture; increasing the number of hidden units or layers makes this bulk sharper and larger, reflecting the inherent over-parametrization.
- The Edges (Spikes): A few large, positive, discrete eigenvalues that stand apart from the bulk. These are primarily determined by the input data. For instance, in a simple 2D two-class problem, exactly two such spikes appear.
This structure implies that the loss landscape is remarkably flat in most directions, with significant curvature existing only in a few dimensions dictated by the data's complexity.
Dynamics of Training
The study observes that training acts as a "singularizer." As optimization progresses, eigenvalues tend to concentrate even more heavily at zero. This suggests that gradient descent naturally finds regions of the landscape that are increasingly flat.
Furthermore, the "flatness" appears to be a global feature rather than a local property of a single minimum. Interpolating between different solutions found by gradient descent reveals that the loss remains low and flat along the entire path.
Conclusion and Future Work
The pervasive singularity of the Hessian in deep learning has significant theoretical implications. Many classical convergence proofs for gradient-based algorithms rely on non-singularity and cannot be directly applied to these models. While this work identifies the two-phase nature of the spectrum, future research is needed to exploit this structure for more efficient optimization. Potential strategies include developing separate approaches for high-curvature "spike" directions and flat "bulk" directions.
Summary updated