Trivializing The Energy Landscape Of Deep Networks

Pratik Chaudhari, Stefano SoattoarXiv 2015 · OpenReview

TL;DR

This paper introduces "AnnealSGD," a novel training algorithm that manipulates the energy landscape of deep neural networks. By drawing parallels to spin glass physics, they show how to transition from a complex landscape with many local minima to a simpler one, leading to faster training and improved generalization.

What Problem Does This Paper Address?

Training deep neural networks is notoriously difficult due to their complex, non-convex energy landscapes, which are riddled with numerous local minima and saddle points. This complexity makes optimization challenging, often leading to issues like vanishing gradients and sensitivity to initialization. While stochastic gradient descent (SGD) is the workhorse, its effectiveness can be hampered by this landscape. The authors aim to understand and exploit the structure of this landscape to develop more efficient and effective training algorithms.

Methods

The core of the paper lies in a theoretical connection between the loss function of sparse deep networks and the Hamiltonian of a spin glass. They leverage this analogy to propose "AnnealSGD." This algorithm introduces a regularization term that acts like a magnetic field, which can be modulated. By starting with a strong "magnetic field," the energy landscape is simplified from one with exponentially many local minima to one with a polynomial number, and eventually to a trivial landscape with a single minimum. As training progresses, this "field" is gradually reduced, guiding the optimization process towards the original, more complex landscape but in a more controlled manner. This annealing process is designed to alleviate issues like vanishing gradients and improve generalization.

Datasets

  • MNIST: Used for training and evaluating fully-connected and LeNet-style convolutional neural networks.
  • CIFAR-10: Used for training and evaluating a modified All-CNN-C network.

Key Insights and Ideas

  • Deep Networks as Spin Glasses: The paper establishes a compelling theoretical link between the loss landscape of sparse deep networks and the well-studied physics of spin glasses. This analogy provides a powerful framework for understanding the optimization challenges.
  • Landscape Trivialization: The authors demonstrate that by adding a specific type of regularization (akin to a magnetic field in spin glasses), the complex energy landscape of a deep network can be systematically simplified. This transition from exponential to polynomial to a single minimum is a key theoretical contribution.
  • Annealing for Better Optimization: AnnealSGD leverages this landscape manipulation. By starting in a "relaxed" polynomial regime and gradually annealing the regularization, the algorithm aims to escape difficult regions of the landscape and find better minima, leading to improved performance.
  • Gradient Noise vs. Perturbation Direction: The paper argues that while gradient noise can help escape saddle points, fixing the direction of the perturbation (as in AnnealSGD) is more beneficial than resampling it at each step, as it provides a consistent bias.

Key Results

  • Faster Training: For a fully-connected network, AnnealSGD showed dramatically faster training in the beginning compared to standard ADAM, with a significantly larger minimum absolute value of the back-propagated gradient.
  • Improved Generalization:
    • For the fully-connected network, AnnealSGD achieved a slightly lower final validation error (2.48%) compared to the baseline (2.92%).
  • For a LeNet-style network on MNIST, AnnealSGD achieved a validation error of 0.55 ± 0.03%, compared to 0.61 ± 0.04% without annealing.
  • On CIFAR-10, an All-CNN-BN network trained with AnnealSGD achieved a test error of 7.95%, outperforming the baseline (8.42%) and other competitive networks.
  • Theoretical Guarantees: The paper provides theoretical analysis showing how the number of critical points and local minima changes with the perturbation, supporting the idea of landscape simplification.

Limitations and Future Work

  • Theoretical Model vs. Modern Architectures: The theoretical model used (sparse, fully-connected or simple convolutional layers) is a simplification of modern, complex deep networks with feature maps and advanced regularization. The authors acknowledge this and use heuristics to adapt their approach.
  • Spin Glass Analogy: While insightful, the spin glass analogy is primarily for understanding the ground state thermodynamics. The dynamics of SGD in complex landscapes, especially concerning saddle points, is a separate, challenging area.
  • Future Work: The authors suggest further study into the landscape properties of convolutional architectures, potentially using spin glasses as inspiration. They also propose investigating whether the inherent gradient noise of SGD has effects similar to an external magnetic field, which could inform strategies for batch size selection and data augmentation.

Summary updated

Abstract

We introduce "AnnealSGD", a regularized stochastic gradient descent algorithm motivated by an analysis of the energy landscape of a particular class of deep networks with sparse random weights. The loss function of such networks can be approximated by the Hamiltonian of a spherical spin glass with Gaussian coupling. While different from currently-popular architectures such as convolutional ones, spin glasses are amenable to analysis, which provides insights on the topology of the loss function and motivates algorithms to minimize it. Specifically, we show that a regularization term akin to a magnetic field can be modulated with a single scalar parameter to transition the loss function from a complex, non-convex landscape with exponentially many local minima, to a phase with a polynomial number of minima, all the way down to a trivial landscape with a unique minimum. AnnealSGD starts training in the relaxed polynomial regime and gradually tightens the regularization parameter to steer the energy towards the original exponential regime. Even for convolutional neural networks, which are quite unlike sparse random networks, we empirically show that AnnealSGD improves the generalization error using competitive baselines on MNIST and CIFAR-10.
BibTeX
@misc{chaudhari2015trivializing,
  author = {Pratik Chaudhari and Stefano Soatto},
  title = {{Trivializing The Energy Landscape Of Deep Networks}},
  year = {2015},
  eprint = {1511.06485},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1511.06485},
}
Figure