Corridor Geometry in Gradient-Based Optimization

Benoit Dherin, Mihaela RoscaarXiv 2024 · arXiv

TL;DR

This paper identifies "corridors" in neural network loss landscapes—regions where the path of steepest descent is a straight line. In these corridors, discrete gradient descent perfectly tracks its continuous-time ideal, enabling a simple adaptive learning rate (CLR) that achieves rapid convergence without the usual discretization-induced instabilities.

What Problem Does This Paper Address?

Deep learning optimization usually relies on Gradient Descent (GD), which is a discrete approximation of continuous Gradient Flow (GF). The gap between the two, known as discretization drift, is a double-edged sword: while it can provide beneficial "implicit regularization," it is also a primary source of training instabilities.

The authors aim to characterize the specific geometry where this drift vanishes. By understanding these "corridors," researchers can better predict when implicit regularization will occur and design optimizers that exploit the local geometry of the loss surface.

Methods

The authors define corridors as regions of a loss surface where the solutions of the gradient flow are straight lines traveled at constant speed. They prove a key analytical result: a region is a corridor if and only if the product of the loss Hessian ($H$) and the gradient ($g$) vanishes ($Hg = 0$).

Based on this geometric insight, they derive the Corridor Learning Rate (CLR): $$h(\theta) = \frac{E(\theta)}{\|g(\theta)\|^2}$$ This adaptive step size is designed to reach the global minimum ($E=0$) in a single step if the optimizer is inside a corridor. The authors show that CLR is a special case of the classical Polyak step-size, providing a new geometric justification for an algorithm originally developed for non-smooth convex optimization.

Datasets

The researchers evaluated the effectiveness of the CLR across several standard computer vision benchmarks:

  • CIFAR-10: Tested with ResNet-18, ResNet-50, and VGG architectures.
  • ImageNet: Tested with ResNet-50 to demonstrate scalability to large-scale problems.

Key Insights and Ideas

  • The $Hg=0$ Criterion: This simple mathematical condition identifies where discrete steps perfectly align with continuous flow. Geometrically, these regions are "ruled surfaces," such as planes or cones. Visualizing corridors as ruled surfaces where steepest descent follows straight lines.

  • Vanishing Implicit Regularization: Inside corridors, there are no implicit regularization effects derived from discretization error because the error itself is zero. This highlights that regularization is a property of the path taken, not just the loss function.

  • Universal Alignment: The paper proves that not only GD, but also momentum-based methods and higher-order integrators like Runge-Kutta 4, follow the GF trajectory exactly within these corridors.
  • Automated Scheduling: CLR naturally produces a learning rate schedule that includes an initial "warm-up" followed by a decay, mimicking the manually tuned schedules often used by practitioners.
Figure 1: Visualizing corridors. (a): Ruled surfaces forming a corridor, with example lines of steepest descent. (b): A line of steepest descent (shown in red) formed on a loss surface constructed from multiple corridors.

Key Results

  • Superior Convergence: On CIFAR-10, CLR reached zero training loss significantly faster than SGD with a fixed learning rate, even after the fixed rate was heavily tuned via grid search. ResNet-18 on CIFAR-10: CLR (red dashed line) converges faster than various fixed learning rates.

  • Robustness to Batch Size: CLR demonstrated stable training on ImageNet across a wide range of batch sizes (from 1024 to 8192) without requiring manual hyperparameter adjustments for each configuration.

  • Dynamic Adaptation: The learning rate automatically peaks early in training and decays as the model approaches a minimum, providing a principled way to handle the "early phase" of training. CLR values over time show an automated warm-up and decay pattern.
Figure 2: The adaptive CLR converges for a ResNet-18 trained on CIFAR-10, and it does so quicker than SGD with a fixed learning rate obtained from a sweep. Batch size 4096. Results across a wide range of batch sizes can be found in Figure 9. Results obtained using 3 seeds.
CLR values over time show an automated warm-up and decay pattern.

Limitations and Future Work

The authors acknowledge that "exact" corridors (where $Hg$ is exactly zero) are unlikely to cover the entire complex landscape of a modern neural network. Instead, corridors serve as a useful theoretical idealization for regions where $Hg$ is small.

The current CLR formulation also assumes the optimal loss value is zero, which is common in over-parameterized networks but may not hold for all tasks. Future work could explore how "near-corridor" regions influence the emergence of implicit regularization and whether these geometric properties can be used to detect or prevent training instabilities in real-time.

Summary updated

Abstract

We characterize regions of a loss surface as corridors when the continuous curves of steepest descent -- the solutions of the gradient flow -- become straight lines. We show that corridors provide insights into gradient-based optimization, since corridors are exactly the regions where gradient descent and the gradient flow follow the same trajectory, while the loss decreases linearly. As a result, inside corridors there are no implicit regularization effects or training instabilities that have been shown to occur due to the drift between gradient descent and the gradient flow. Using the loss linear decrease on corridors, we devise a learning rate adaptation scheme for gradient descent; we call this scheme Corridor Learning Rate (CLR). The CLR formulation coincides with a special case of Polyak step-size, discovered in the context of convex optimization. The Polyak step-size has been shown recently to have also good convergence properties for neural networks; we further confirm this here with results on CIFAR-10 and ImageNet.
BibTeX
@misc{dherin2024corridor,
  author = {Benoit Dherin and Mihaela Rosca},
  title = {{Corridor Geometry in Gradient-Based Optimization}},
  year = {2024},
  eprint = {2402.08818},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.08818},
}
Figure