A Deeper Look at the Hessian Eigenspectrum of Deep Neural Networks and its Applications to Regularization

Adepu Ravi Sankar, Yash Khasbage, Rahul Vigneswaran, Vineeth N. Balasubramanian

Abstract

Loss landscape analysis is extremely useful for a deeper understanding of the generalization ability of deep neural network models. In this work, we propose a layerwise loss landscape analysis where the loss surface at every layer is studied independently and also on how each correlates to the overall loss surface. We study the layerwise loss landscape by studying the eigenspectra of the Hessian at each layer. In particular, our results show that the layerwise Hessian geometry is largely similar to the entire Hessian. We also report an interesting phenomenon where the Hessian eigenspectrum of middle layers of the deep neural network are observed to most similar to the overall Hessian eigenspectrum. We also show that the maximum eigenvalue and the trace of the Hessian (both full network and layerwise) reduce as training of the network progresses. We leverage on these observations to propose a new regularizer based on the trace of the layerwise Hessian. Penalizing the trace of the Hessian at every layer indirectly forces Stochastic Gradient Descent to converge to flatter minima, which are shown to have better generalization performance. In particular, we show that such a layerwise regularizer can be leveraged to penalize the middlemost layers alone, which yields promising results. Our empirical studies on well-known deep nets across datasets support the claims of this work.

Introduction

Understanding the "curvature" of a deep neural network's loss landscape is critical for improving generalization. A long-standing hypothesis in deep learning is that flat minima—regions of the loss landscape with gentle slopes—generalize better to unseen data than sharp minima. While the curvature of the entire network (quantified by the Hessian matrix) has been studied, it is computationally expensive and treats the model as a "black box."

Researchers from IIT Hyderabad address this by proposing a layerwise analysis. They investigate how each layer contributes to the overall landscape and whether specific layers dominate the network's geometric properties. Their findings lead to the introduction of Hessian Trace Regularization (HTR), a method that penalizes layerwise curvature to steer models toward flatter minima.

Methodology

The study employs several numerical linear algebra techniques to analyze the Hessian:

  1. Hessian Eigenspectrum Analysis: Studying the distribution of eigenvalues for individual layers. Large eigenvalues indicate high curvature (sharpness).
  2. Gauss-Newton Decomposition: Demonstrating that significant Hessian structure arises from the gradient-related "G" term, while the second-order "H" term is negligible.
  3. Efficient Estimation: Using the Lanczos method for top eigenvalues and the Hutchinson method to estimate the Trace (sum of eigenvalues) without computing the full matrix.
  4. Hessian Trace Regularization (HTR): A new loss function: $\text{Loss} = \text{CrossEntropy} + \gamma \sum \text{Tr}(\text{Hessian}_l)$, which explicitly penalizes curvature.

Key Insights

The "Middle Layer" Soul

A striking discovery is that the Hessian eigenspectrum of the middle layers is the most representative of the entire network's spectrum. Quantitative analysis shows that the distance between the layerwise and full Hessian spectra reaches a minimum at the middle of the network depth across various architectures and datasets.

Distance between layerwise and full Hessian spectra across network depth.

Natural Flattening and Class Signatures

In every layer, the number of "outlier" eigenvalues is approximately equal to the number of classes ($C$), suggesting discriminative capability is distributed throughout the network. Furthermore, as training progresses, the maximum eigenvalue and the trace naturally decrease, a process accelerated by architectural choices like Batch Normalization.

Evolution of the Hessian trace and maximum eigenvalue during training.

Empirical Results

  • Improved Generalization: HTR consistently reduces generalization error. For instance, a VGG11 model on CIFAR-10 saw error drop from 18.20% to 15.11%.
  • Synergy with Weight Decay: HTR complements standard L2 regularization, indicating it captures distinct aspects of model complexity.
  • Targeted Regularization: Regularizing only the middle layers proved nearly as effective as full-network regularization, offering a more computationally efficient path.

Discussion and Future Work

While HTR improves performance, calculating Hessian traces remains more expensive than standard backpropagation. Future work involves finding even faster approximations and exploring adaptive weighting schemes for different layers. Additionally, while the "middle layer" phenomenon is empirically robust, a rigorous mathematical proof for why these layers dominate the geometric identity remains an open question.

Summary updated

Abstract

Loss landscape analysis is extremely useful for a deeper understanding of the generalization ability of deep neural network models. In this work, we propose a layerwise loss landscape analysis where the loss surface at every layer is studied independently and also on how each correlates to the overall loss surface. We study the layerwise loss landscape by studying the eigenspectra of the Hessian at each layer. In particular, our results show that the layerwise Hessian geometry is largely similar to the entire Hessian. We also report an interesting phenomenon where the Hessian eigenspectrum of middle layers of the deep neural network are observed to most similar to the overall Hessian eigenspectrum. We also show that the maximum eigenvalue and the trace of the Hessian (both full network and layerwise) reduce as training of the network progresses. We leverage on these observations to propose a new regularizer based on the trace of the layerwise Hessian. Penalizing the trace of the Hessian at every layer indirectly forces Stochastic Gradient Descent to converge to flatter minima, which are shown to have better generalization performance. In particular, we show that such a layerwise regularizer can be leveraged to penalize the middlemost layers alone, which yields promising results. Our empirical studies on well-known deep nets across datasets support the claims of this work.
BibTeX
@inproceedings{sankar2021deeper,
  author = {Adepu Ravi Sankar and Yash Khasbage and Rahul Vigneswaran and Vineeth N. Balasubramanian},
  title = {{A Deeper Look at the Hessian Eigenspectrum of Deep Neural Networks and its Applications to Regularization}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=CewvowuBiDk},
}
Figure