Second Order Properties of Error Surfaces

Yann LeCun, Ido Kanter, Sara A. SollaNeurIPS 1990 · OpenReview

TL;DR

This paper analytically investigates the "error surface" of neural networks by examining the Hessian matrix's eigenvalue spectrum. This analysis reveals how the shape of the error surface dictates learning speed and provides theoretical backing for practical techniques to accelerate training, particularly for linear units and offering insights into multi-layer networks.

What Problem Does This Paper Address?

Training neural networks often involves navigating a complex, high-dimensional "error surface" to find the optimal set of weights (parameters) that minimize a cost function. The speed and efficiency of this search are heavily influenced by the local geometry of this surface. Gradient descent, a common training algorithm, can be slow if the surface is highly elongated or has very different curvatures along different directions. This paper aims to understand these geometric properties, specifically the "second-order properties" (related to curvature), to explain and improve learning dynamics. The authors focus on understanding the time scales involved in reaching the optimal solution and identifying factors that lead to slow convergence.

Methods

The core of this paper is an analytical investigation of the Hessian matrix of the cost function. For a simplified model of a single linear neuron, the cost function is quadratic, allowing for an exact calculation of the Hessian, which in this case is the input covariance matrix. The authors then analyze the eigenvalue spectrum of this covariance matrix. Eigenvalues of the Hessian correspond to the curvatures along different directions in the parameter space. By understanding the distribution of these eigenvalues, they can characterize the shape of the error surface. They use a coordinate transformation to align with the principal axes of the error surface, simplifying the analysis of the learning dynamics. This theoretical framework is then used to derive learning time estimates and explore the impact of various factors like input statistics and training set size.

Datasets

Not discussed in the paper. The paper focuses on theoretical analysis and uses synthetic data distributions for its derivations and simulations. Figure 2 shows experimental results for training a single linear neuron with randomly generated inputs, but specific dataset names are not provided.

Key Insights and Ideas

  • Eigenvalue Spectrum as a Learning Speedometer: The distribution of eigenvalues of the Hessian matrix directly dictates the learning time. A wide spread of eigenvalues (large ratio of max to min eigenvalue) implies slow convergence, as the learning algorithm must contend with directions of very different curvatures.
  • The "Slowest Mode" Problem: The slowest convergence is determined by the smallest non-zero eigenvalue of the Hessian. This "slowest mode" often dictates the overall learning time.
  • Centering Inputs is Crucial: Biased inputs (non-zero mean) can lead to a very large eigenvalue, significantly slowing down learning. Centering the input variables (subtracting their mean) is a theoretically justified method to mitigate this and accelerate convergence.
  • Optimal Training Set Size: Surprisingly, for a single linear neuron, there exists an optimal ratio of training examples to input dimensions (represented by 'a') that minimizes learning time, rather than simply more data always being better.
  • Justification for Sigmoid Choices: The paper provides a theoretical explanation for why symmetric activation functions (like tanh) often lead to faster convergence than non-symmetric ones (like the logistic function), linking it back to the eigenvalue spectrum and the issue of biased inputs.

Key Results

  • For a single linear unit, the learning time $\tau$ is proportional to the ratio of the maximum to minimum non-zero eigenvalues of the input covariance matrix, $\tau \propto \alpha (\lambda_{max}/\lambda_{min})$.
  • The paper derives an analytical expression for the learning time $\tau$ as a function of the training set size to input dimension ratio $\alpha$, showing a divergence at $\alpha=1$ and a minimum around $\alpha \approx 5.83$ for centered inputs.
  • Numerical simulations (Figure 2) confirm the theoretical prediction of an optimal training set size for faster learning.
  • The analysis demonstrates that for centered inputs (zero mean), the eigenvalue spectrum is well-behaved, leading to faster convergence compared to biased inputs.

Limitations and Future Work

  • Focus on Linear Units: While the paper discusses generalization to multi-layer nonlinear networks, the rigorous analytical results are primarily derived for single linear units. Extending these findings to complex non-linear architectures remains a challenge.
  • Assumptions on Input Distribution: The analysis of the eigenvalue spectrum relies on assumptions about the statistical properties of the input data (e.g., independence, variance). Real-world data may deviate from these assumptions.
  • Future Work: The authors suggest that their theoretical framework can be used to develop novel approaches for accelerating learning in multi-layer networks. This includes exploring adaptive learning rates for individual neurons, proportional to the inverse of their connectivity, and using centered state variables. Further investigation into the impact of correlated inputs on the eigenvalue spectrum is also implied.

Summary updated

Abstract

The learning time of a simple neural network model is obtained through an analytic computation of the eigenvalue spectrum for the Hessian matrix, which describes the second order properties of the cost function in the space of coupling coefficients. The form of the eigenvalue distribution suggests new techniques for accelerating the learning process, and provides a theoretical justification for the choice of centered versus biased state variables.
BibTeX
@inproceedings{lecun1990second,
  author = {Yann LeCun and Ido Kanter and Sara A. Solla},
  title = {{Second Order Properties of Error Surfaces}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1990},
  url = {https://openreview.net/forum?id=rJVrpOb_WH},
}

Loading related content...

Figure