The Loss Surface of Multilayer Networks

Anna Choromanska, Mikael Henaff, Michaël Mathieu, Gérard Ben Arous, Yann LeCun

Abstract

We study the connection between the highly non-convex loss function of a simple model of the fully-connected feed-forward neural network and the Hamiltonian of the spherical spin-glass model. Under assumptions of variable independence, redundancy in network parametrization, and uniformity, we leverage results from random matrix theory to explain the complexity of decoupled neural networks. We show that for large networks, the lowest critical values of the random loss function form a layered structure in a well-defined band lower-bounded by the global minimum. The number of local minima outside this band diminishes exponentially with network size. Empirically, we verify that real networks exhibit similar behavior. We conjecture that optimization algorithms like SGD converge to this band of high-quality local minima, while the global minimum becomes harder to reach and often leads to overfitting.

Introduction: The Optimization Paradox

In the early days of deep learning, the non-convexity of loss surfaces was seen as a major obstacle. Conventional wisdom suggested that optimizers would frequently get trapped in "bad" local minima with high error. However, modern practice reveals a paradox: as networks grow larger, they become easier to train, and different initializations consistently lead to similar, high-quality performance.

This paper addresses the fundamental geometry of the loss surface. Why do we avoid catastrophic local minima? And why is the global minimum not the ultimate goal?

Theoretical Framework: The Spin-Glass Connection

The authors bridge neural network theory and physics by mapping a simplified, "fully decoupled" ReLU network to a high-degree polynomial. Under three key assumptions—variable independence, parameter redundancy, and uniformity—the loss function becomes mathematically analogous to the Hamiltonian of a spherical spin-glass model, a system used in physics to describe disordered magnetic systems.

Theoretical complexity functions describing the distribution of critical points.

By applying proofs from spin-glass theory, the authors analyze the distribution of critical points (maxima, minima, and saddle points). The complexity function $\Theta_H(u)$ (shown in Figure 1) characterizes the exponential growth of the number of critical points with the system size. The function $\Theta_{k,H}(u)$ specifically tracks critical points of index $k$, where $k$ is the number of negative eigenvalues of the Hessian.

Key Insights into Loss Geometry

1. The "Good" Band

For large networks, the lowest-value critical points are not isolated. Instead, they form a layered "band" just above the global minimum. Most local minima found by algorithms like Stochastic Gradient Descent (SGD) reside within this band, offering excellent generalization.

2. Saddle Points vs. Local Minima

The theory suggests that "bad" (high-value) critical points are almost exclusively saddle points, not local minima. These points have many "downward" directions, allowing optimizers like SGD to escape them easily. As the index $k$ increases, the critical points become more numerous but less stable, effectively acting as "passages" rather than "traps."

3. The Effect of Network Size

As network size increases, the probability of encountering a "bad" local minimum far from the global minimum decreases exponentially. This explains why larger models are more robust to initialization and easier to optimize.

Distribution of critical points showing concentration in a "good" band.

Figure 6 illustrates how critical points concentrate. The plots show that the mean number of critical points peaks sharply in a specific energy range. The zoomed views demonstrate that low-index critical points (potential local minima) are concentrated in a narrow energy band near the bottom of the landscape, which the authors term the "good" band.

4. The Irrelevance of the Global Minimum

The authors prove that recovering the absolute global minimum becomes harder as the network grows. More importantly, they argue it is practically irrelevant: the global minimum often corresponds to overfitting the training data, whereas the "good" local minima in the band generalize better to unseen data.

Empirical Validation

The authors tested these theoretical predictions using synthetic spin-glass simulations and experiments on MNIST. For the MNIST tests, images were downsampled to 10x10 pixels to make the computation of the Hessian matrix feasible.

Key Results: * Loss Concentration: As the number of hidden units increases, the distribution of test losses becomes highly concentrated. The variance in performance between different runs nearly vanishes for large networks. * Low-Index Solutions: By calculating the eigenvalues of the Hessian at the end of training, the authors confirmed that SGD consistently finds solutions that are either true local minima or saddle points of very low index. * Train-Test Decorrelation: In larger networks, the correlation between training and test loss was found to decrease (dropping from 0.76 to 0.40 in their experiments). This supports the idea that obsessing over the lowest possible training loss (the global minimum) does not necessarily yield better test performance.

Conclusion and Limitations

The primary limitation is the independence assumption. Real neural networks have highly dependent paths due to shared weights and inputs. While empirical results suggest the spin-glass analogy holds even in the presence of these dependencies, a rigorous mathematical proof remains an open challenge. Future work may extend this analysis to more complex architectures like Convolutional Neural Networks (CNNs) or Recurrent Neural Networks (RNNs), where structural constraints might further shape the loss landscape.

Summary updated

Abstract

We study the connection between the highly non-convex loss function of a simple model of the fully-connected feed-forward neural network and the Hamiltonian of the spherical spin-glass model under the assumptions of: i) variable independence, ii) redundancy in network parametrization, and iii) uniformity. These assumptions enable us to explain the complexity of the fully decoupled neural network through the prism of the results from random matrix theory. We show that for large-size decoupled networks the lowest critical values of the random loss function form a layered structure and they are located in a well-defined band lower-bounded by the global minimum. The number of local minima outside that band diminishes exponentially with the size of the network. We empirically verify that the mathematical model exhibits similar behavior as the computer simulations, despite the presence of high dependencies in real networks. We conjecture that both simulated annealing and SGD converge to the band of low critical points, and that all critical points found there are local minima of high quality measured by the test error. This emphasizes a major difference between large- and small-size networks where for the latter poor quality local minima have non-zero probability of being recovered. Finally, we prove that recovering the global minimum becomes harder as the network size increases and that it is in practice irrelevant as global minimum often leads to overfitting.
BibTeX
@inproceedings{choromanska2015loss,
  author = {Anna Choromanska and Mikael Henaff and Michaël Mathieu and Gérard Ben Arous and Yann LeCun},
  title = {{The Loss Surface of Multilayer Networks}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2015},
  url = {https://openreview.net/forum?id=HTCSSGUhqT},
}
Figure