The Spectrum of the Fisher Information Matrix of a Single-Hidden-Layer Neural Network

Jeffrey Pennington, Pratik WorahNeurIPS 2018 · OpenReview

Introduction

Deep learning’s remarkable success relies heavily on the ability to optimize massive networks with millions or billions of parameters using simple first-order methods like Stochastic Gradient Descent (SGD). However, the "geometry" of the loss surfaces these algorithms navigate remains largely a black box.

The speed and efficiency of optimization are dictated by the local curvature of the loss surface, typically described by the Hessian matrix or its close relative, the Fisher Information Matrix (FIM). Because these matrices are gargantuan in modern networks, it is computationally impossible to study their full spectra (the distribution of their eigenvalues). Consequently, the field has lacked a precise understanding of how architectural choices—such as the selection of activation functions—actually alter the "valleys" and "hills" traversed by the optimizer. This paper bridges that gap by providing a rigorous mathematical framework to predict these spectra for single-hidden-layer neural networks.

Methodology: Random Matrix Theory

To tackle this problem, the authors utilize Random Matrix Theory (RMT) to analyze a single-hidden-layer network with $n$ inputs, $n$ hidden units, and $n$ outputs. The theoretical setup assumes the input data and weights are independent and identically distributed (i.i.d.) Gaussian variables.

The analytical approach relies on several key technical steps:

  • The Moment Method: Calculating the traces of high powers of the FIM to understand its underlying distribution.
  • Stieltjes Transform: Converting those moments into a continuous density function that represents the eigenvalue spectrum.
  • Gauss-Newton Approximation: For squared loss, the FIM is equivalent to the Gauss-Newton matrix, allowing the analysis to focus squarely on the Jacobian of the network's output.

Key Insights: The Healing Power of Nonlinearity

One of the most striking findings of the paper is the stark contrast in the loss surface geometry between linear and nonlinear networks.

Comparison of FIM spectra for linear vs. nonlinear networks
  • Linearity is Pathological: Surprisingly, linear networks (where $f(x)=x$) are highly poorly conditioned. They exhibit a massive number of zero eigenvalues, which translates to "flat" directions in the loss surface where an optimizer can easily get stuck.
  • Nonlinearities "Heal" the Spectrum: Introducing a nonlinearity (such as ReLU or the error function, Erf) generically removes this degeneracy. It spreads the eigenvalues away from zero, introducing curvature that helps gradient descent find a productive direction.

To quantify this, the authors propose using the ratio of the second moment to the square of the first moment ($m_2/m_1^2$) as a measure of conditioning. A lower ratio indicates that the eigenvalues are more tightly clustered, which is theoretically ideal for optimization. Taking this a step further, they derive an "optimal" activation function ($f_{opt}$) specifically designed to minimize this ratio, proving that activation functions can be mathematically engineered to ease optimization.

Empirical Validation and Results

The paper's theoretical heavy lifting culminates in an exact analytical expression—an explicit integral formula for the FIM spectrum based on constants derived from the chosen activation function.

To validate these infinite-width theoretical predictions, the authors ran simulations using finite-width networks.

Empirical validation of the FIM spectrum across various network widths

As shown in the experiments, networks with a width of 128 showed nearly perfect agreement with the theoretical predictions for infinite-width networks. Furthermore, in a "student-teacher" optimization setup, activation functions predicted to have better conditioning (a lower $m_2/m_1^2$ ratio) consistently achieved greater loss reduction in a single gradient step, directly linking the spectral theory to practical optimization speedups.

Limitations and Future Work

While groundbreaking, this theoretical framework has limitations. The analysis is strictly confined to single-hidden-layer networks; extending this to deep architectures is significantly more complex due to how signals propagate through multiple layers. Additionally, the reliance on i.i.d. Gaussian weights and inputs means the theory may not fully capture the FIM spectrum's behavior on highly structured real-world data like images or text. Finally, the theoretically "optimal" activation function grows very quickly for large inputs, which could trigger exploding gradients in practical training scenarios.

Despite these constraints, this work lays a crucial foundation. The RMT framework introduced here opens the door to studying more complex architectures and potentially understanding the "training dynamics"—how the spectrum of the loss surface evolves as a neural network learns.

Summary updated

Abstract

An important factor contributing to the success of deep learning has been the remarkable ability to optimize large neural networks using simple first-order optimization algorithms like stochastic gradient descent. While the efficiency of such methods depends crucially on the local curvature of the loss surface, very little is actually known about how this geometry depends on network architecture and hyperparameters. In this work, we extend a recently-developed framework for studying spectra of nonlinear random matrices to characterize an important measure of curvature, namely the eigenvalues of the Fisher information matrix. We focus on a single-hidden-layer neural network with Gaussian data and weights and provide an exact expression for the spectrum in the limit of infinite width. We find that linear networks suffer worse conditioning than nonlinear networks and that nonlinear networks are generically non-degenerate. We also predict and demonstrate empirically that by adjusting the nonlinearity, the spectrum can be tuned so as to improve the efficiency of first-order optimization methods.
BibTeX
@inproceedings{pennington2018spectrum,
  author = {Jeffrey Pennington and Pratik Worah},
  title = {{The Spectrum of the Fisher Information Matrix of a Single-Hidden-Layer Neural Network}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=SJW_kvZ_WB},
}
Figure