Hessian Eigenspectra of More Realistic Nonlinear Models

Zhenyu Liao, Michael W. Mahoney

Abstract

The Hessian matrix and its eigenspectrum are critical for analyzing optimization landscapes, yet traditional analyses often rely on oversimplified assumptions. This paper introduces a precise characterization of Hessian eigenspectra for Generalized Generalized Linear Models (G-GLMs) using deterministic equivalent techniques from Random Matrix Theory. We demonstrate that the interplay between data statistics, model structure, and loss functions generates complex spectral features—including multiple bulks and isolated spikes—that align perfectly with empirical observations.

Introduction

In high-dimensional optimization, the Hessian matrix serves as a map of the local curvature, identifying flat valleys, sharp peaks, and saddle points. While researchers frequently apply Random Matrix Theory (RMT) to study these spectra, they often assume the Hessian follows simple distributions like the Marčenko–Pastur law. However, empirical studies of neural networks reveal "spikes" and "heavy tails" that these toy models cannot explain. This work addresses this gap by investigating how specific properties of data, model architecture, and loss functions shape the Hessian's spectrum in realistic nonlinear settings.

Theoretical Framework: G-GLMs and Deterministic Equivalents

The authors focus on Generalized Generalized Linear Models (G-GLMs), a flexible family covering logistic regression, phase retrieval, and single-layer neural networks. The core innovation lies in applying deterministic equivalent techniques to handle the nonlinear dependencies within the Hessian. Unlike standard RMT, which assumes independent components, this approach accounts for second derivatives that depend nonlinearly on the data.

The framework establishes two primary components:

  1. Limiting Spectral Measure: Predicting the "bulk" of the eigenvalue distribution as dimensions $p$ and $n$ scale to infinity.
  2. Isolated Eigenpairs: Characterizing "spikes" that emerge outside the bulk and the alignment of their eigenvectors with true model parameters.
Empirical vs. Theoretical Hessian Eigenspectra: (a, b) Logistic models with different weight initializations; (c) Phase retrieval model showing dual spikes; (d) Alignment of the top eigenvector with true parameters.

Key Insights and Results

The analysis reveals that the Hessian spectrum is an emergent property of a "nontrivial interplay" between several factors:

  • Dual Origins of Spikes: The authors identify two distinct types of isolated eigenvalues. "Signal spikes" are driven by the data's mean, while "model spikes" arise purely from the model's nonlinear structure, even without a data signal.
  • The Role of Loss Functions: A striking finding is that the loss function can qualitatively transform the spectrum. For the same logistic model, a logistic loss yields a bounded spectrum, while an exponential loss creates an unbounded, heavy-tailed distribution.
  • Multi-bulk and Curvature: If input data has heterogeneous features, the Hessian can split into multiple disconnected "bulks." Furthermore, model-driven spikes can appear on the left side of the bulk, representing directions of significantly lower curvature.
  • Eigenvector Alignment: In non-convex scenarios like phase retrieval, the top Hessian eigenvector correlates strongly with the true parameters $\mathbf{w}_*$. This provides a theoretical justification for "spectral initialization" methods used in practice.

Conclusion

By moving beyond oversimplified RMT models, this research unveils the theoretical origins of striking features observed in complex machine learning models. While currently focused on single-layer architectures, the deterministic equivalent approach offers a promising path toward understanding the heavy-tailed correlations and generalization properties of deep neural networks.

Summary updated

Abstract

Given an optimization problem, the Hessian matrix and its eigenspectrum can be used in many ways, ranging from designing more efficient second-order algorithms to performing model analysis and regression diagnostics. When nonlinear models and non-convex problems are considered, strong simplifying assumptions are often made to make Hessian spectral analysis more tractable. This leads to the question of how relevant the conclusions of such analyses are for more realistic nonlinear models. In this paper, we exploit deterministic equivalent techniques from random matrix theory to make a \emph{precise} characterization of the Hessian eigenspectra for a broad family of nonlinear models, including models that generalize the classical generalized linear models, without relying on strong simplifying assumptions used previously. We show that, depending on the data properties, the nonlinear response model, and the loss function, the Hessian can have \emph{qualitatively} different spectral behaviors: of bounded or unbounded support, with single- or multi-bulk, and with isolated eigenvalues on the left- or right-hand side of the bulk. By focusing on such a simple but nontrivial nonlinear model, our analysis takes a step forward to unveil the theoretical origin of many visually striking features observed in more complex machine learning models.
BibTeX
@inproceedings{liao2021hessian,
  author = {Zhenyu Liao and Michael W. Mahoney},
  title = {{Hessian Eigenspectra of More Realistic Nonlinear Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=6MQ1EweXZj},
}
Figure