Abstract
The Hessian matrix and its eigenspectrum are critical for analyzing optimization landscapes, yet traditional analyses often rely on oversimplified assumptions. This paper introduces a precise characterization of Hessian eigenspectra for Generalized Generalized Linear Models (G-GLMs) using deterministic equivalent techniques from Random Matrix Theory. We demonstrate that the interplay between data statistics, model structure, and loss functions generates complex spectral features—including multiple bulks and isolated spikes—that align perfectly with empirical observations.
Introduction
In high-dimensional optimization, the Hessian matrix serves as a map of the local curvature, identifying flat valleys, sharp peaks, and saddle points. While researchers frequently apply Random Matrix Theory (RMT) to study these spectra, they often assume the Hessian follows simple distributions like the Marčenko–Pastur law. However, empirical studies of neural networks reveal "spikes" and "heavy tails" that these toy models cannot explain. This work addresses this gap by investigating how specific properties of data, model architecture, and loss functions shape the Hessian's spectrum in realistic nonlinear settings.
Theoretical Framework: G-GLMs and Deterministic Equivalents
The authors focus on Generalized Generalized Linear Models (G-GLMs), a flexible family covering logistic regression, phase retrieval, and single-layer neural networks. The core innovation lies in applying deterministic equivalent techniques to handle the nonlinear dependencies within the Hessian. Unlike standard RMT, which assumes independent components, this approach accounts for second derivatives that depend nonlinearly on the data.
The framework establishes two primary components:
- Limiting Spectral Measure: Predicting the "bulk" of the eigenvalue distribution as dimensions $p$ and $n$ scale to infinity.
- Isolated Eigenpairs: Characterizing "spikes" that emerge outside the bulk and the alignment of their eigenvectors with true model parameters.
Key Insights and Results
The analysis reveals that the Hessian spectrum is an emergent property of a "nontrivial interplay" between several factors:
- Dual Origins of Spikes: The authors identify two distinct types of isolated eigenvalues. "Signal spikes" are driven by the data's mean, while "model spikes" arise purely from the model's nonlinear structure, even without a data signal.
- The Role of Loss Functions: A striking finding is that the loss function can qualitatively transform the spectrum. For the same logistic model, a logistic loss yields a bounded spectrum, while an exponential loss creates an unbounded, heavy-tailed distribution.
- Multi-bulk and Curvature: If input data has heterogeneous features, the Hessian can split into multiple disconnected "bulks." Furthermore, model-driven spikes can appear on the left side of the bulk, representing directions of significantly lower curvature.
- Eigenvector Alignment: In non-convex scenarios like phase retrieval, the top Hessian eigenvector correlates strongly with the true parameters $\mathbf{w}_*$. This provides a theoretical justification for "spectral initialization" methods used in practice.
Conclusion
By moving beyond oversimplified RMT models, this research unveils the theoretical origins of striking features observed in complex machine learning models. While currently focused on single-layer architectures, the deterministic equivalent approach offers a promising path toward understanding the heavy-tailed correlations and generalization properties of deep neural networks.
Summary updated